Oct 30 12:48:58.669375 ip-10-0-138-103 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 12:48:58.669388 ip-10-0-138-103 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 12:48:58.669397 ip-10-0-138-103 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 12:48:58.669707 ip-10-0-138-103 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 12:49:08.860016 ip-10-0-138-103 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 12:49:08.860031 ip-10-0-138-103 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 6a9c628e4a2d4cbea29f63493b9a4b06 -- Oct 30 12:51:11.498825 ip-10-0-138-103 systemd[1]: Starting Kubernetes Kubelet... Oct 30 12:51:11.880057 ip-10-0-138-103 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:51:11.880057 ip-10-0-138-103 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 12:51:11.880057 ip-10-0-138-103 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:51:11.880057 ip-10-0-138-103 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:51:11.885668 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885575 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:11.885668 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885665 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:11.885668 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885670 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:11.885668 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885673 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:11.885668 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885676 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885687 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885692 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885695 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885698 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885701 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885704 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885706 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885709 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885712 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885715 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885717 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885720 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885723 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885725 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885728 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885730 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885733 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885736 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885738 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:11.885827 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885741 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885744 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885747 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885749 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885752 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885754 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885757 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885759 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885762 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885765 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885767 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885770 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885773 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885776 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885780 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885782 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885785 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885787 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885790 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:11.886349 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885792 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885795 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885797 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885800 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885803 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885806 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885809 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885811 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885814 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885817 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885819 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885822 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885824 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885827 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885830 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885833 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885836 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885840 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885843 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885845 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:11.886830 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885848 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885851 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885854 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885856 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885859 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885861 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885864 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885867 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885872 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885875 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885878 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885880 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885883 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885886 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885888 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885891 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885894 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885896 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885900 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:11.887306 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885905 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885908 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885911 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885914 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885917 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885919 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885922 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885925 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885928 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885930 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885933 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885936 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885939 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885941 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885945 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885948 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885953 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885956 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885959 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885962 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:11.887767 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885966 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885968 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885971 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.885974 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886843 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886851 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886854 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886856 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886859 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886862 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886865 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886867 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886870 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886872 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886875 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886877 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886880 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886883 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886885 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886888 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:11.888237 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886890 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886892 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886896 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886899 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886902 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886905 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886907 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886910 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886913 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886916 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886919 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886922 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886924 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886927 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886930 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886938 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886941 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886944 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886946 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:11.888766 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886949 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886952 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886957 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886960 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886964 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886967 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886970 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886972 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886975 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886978 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886980 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886983 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886985 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886988 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886990 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886993 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886996 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.886998 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887001 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:11.889221 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887004 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887006 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887009 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887012 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887014 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887017 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887019 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887022 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887025 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887027 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887036 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887038 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887041 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887043 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887047 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887049 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887052 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887054 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887057 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887060 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:11.889698 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887062 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887065 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887067 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887070 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887072 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887075 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887077 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887079 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887082 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887085 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887087 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887090 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887093 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887096 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887098 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887102 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887107 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887110 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887114 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:11.890184 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887117 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887120 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887123 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887125 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887133 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887136 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887139 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887143 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887145 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887148 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887150 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887153 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887155 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887758 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887764 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887768 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887771 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887774 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887777 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:11.890651 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887779 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887782 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887785 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887788 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887791 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887793 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887796 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887799 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887801 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887805 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887808 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887811 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887814 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887817 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887819 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887822 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887825 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887835 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887838 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887841 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:11.891105 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887844 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887846 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887849 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887851 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887854 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887857 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887860 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887863 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887865 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887868 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887871 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887873 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887876 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887879 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887881 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887887 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887889 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887892 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887895 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:11.891600 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887898 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887900 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887903 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887905 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887908 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887910 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887913 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887915 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887918 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887920 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887923 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887933 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887936 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887940 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887944 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887947 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887950 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887954 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887956 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:11.892055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887959 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887962 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887964 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887967 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887969 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887972 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887975 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887977 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887979 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887983 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887986 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887988 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887991 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887993 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887996 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.887998 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888000 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888003 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888005 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888008 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:11.892506 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888011 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888013 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888016 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888019 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888021 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888030 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888033 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888036 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888038 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888041 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888043 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888046 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888049 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888051 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888054 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888056 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888059 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888062 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888065 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:11.892980 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888068 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888070 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.888074 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888798 2572 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888810 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888824 2572 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888829 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888834 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888838 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888842 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888847 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888850 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888853 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888858 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888862 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888865 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888869 2572 flags.go:64] FLAG: --cgroup-root="" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888872 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888875 2572 flags.go:64] FLAG: --client-ca-file="" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888884 2572 flags.go:64] FLAG: --cloud-provider="external" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888887 2572 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888893 2572 flags.go:64] FLAG: --cluster-domain="" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888896 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 12:51:11.893442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888899 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888903 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888906 2572 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888910 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888913 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888916 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888920 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888923 2572 flags.go:64] FLAG: --contention-profiling="false" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888926 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888929 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888932 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888936 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888941 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888944 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888947 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888950 2572 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888953 2572 flags.go:64] FLAG: --enable-server="true" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888956 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888963 2572 flags.go:64] FLAG: --event-burst="100" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888966 2572 flags.go:64] FLAG: --event-qps="50" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888970 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888973 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888976 2572 flags.go:64] FLAG: --eviction-hard="" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888981 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 12:51:11.894038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888984 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888987 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888990 2572 flags.go:64] FLAG: --eviction-soft="" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888993 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.888996 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889007 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889010 2572 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889013 2572 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889016 2572 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889019 2572 flags.go:64] FLAG: --feature-gates="" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889023 2572 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889026 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889029 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889033 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889036 2572 flags.go:64] FLAG: --healthz-port="10248" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889039 2572 flags.go:64] FLAG: --help="false" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889042 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-138-103.ec2.internal" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889045 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889048 2572 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889051 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889055 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889058 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889061 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 12:51:11.894684 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889064 2572 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889067 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889070 2572 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889073 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889080 2572 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889083 2572 flags.go:64] FLAG: --kube-reserved="" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889086 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889089 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889092 2572 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889095 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889098 2572 flags.go:64] FLAG: --lock-file="" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889101 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889104 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889107 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889112 2572 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889121 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889124 2572 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889127 2572 flags.go:64] FLAG: --logging-format="text" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889130 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889134 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889137 2572 flags.go:64] FLAG: --manifest-url="" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889140 2572 flags.go:64] FLAG: --manifest-url-header="" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889145 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889148 2572 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889152 2572 flags.go:64] FLAG: --max-pods="110" Oct 30 12:51:11.895236 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889155 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889158 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889161 2572 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889164 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889167 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889170 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889173 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889180 2572 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889183 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889186 2572 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889190 2572 flags.go:64] FLAG: --pod-cidr="" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889194 2572 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889197 2572 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889200 2572 flags.go:64] FLAG: --pods-per-core="0" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889203 2572 flags.go:64] FLAG: --port="10250" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889206 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889209 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0adfb01818ff2b9e7" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889212 2572 flags.go:64] FLAG: --qos-reserved="" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889216 2572 flags.go:64] FLAG: --read-only-port="10255" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889218 2572 flags.go:64] FLAG: --register-node="true" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889221 2572 flags.go:64] FLAG: --register-with-taints="" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889226 2572 flags.go:64] FLAG: --registry-burst="10" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889229 2572 flags.go:64] FLAG: --registry-qps="5" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889238 2572 flags.go:64] FLAG: --reserved-cpus="" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889242 2572 flags.go:64] FLAG: --reserved-memory="" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889246 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 12:51:11.895947 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889249 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889252 2572 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889255 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889258 2572 flags.go:64] FLAG: --runonce="false" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889261 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889265 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889268 2572 flags.go:64] FLAG: --seccomp-default="false" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889271 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889274 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889277 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889281 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889284 2572 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889287 2572 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889290 2572 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889293 2572 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889296 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889299 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889303 2572 flags.go:64] FLAG: --system-cgroups="" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889307 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889312 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889315 2572 flags.go:64] FLAG: --tls-cert-file="" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889318 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889324 2572 flags.go:64] FLAG: --tls-min-version="" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889327 2572 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889330 2572 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 12:51:11.897063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889333 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889336 2572 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889339 2572 flags.go:64] FLAG: --v="2" Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889344 2572 flags.go:64] FLAG: --version="false" Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889349 2572 flags.go:64] FLAG: --vmodule="" Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889361 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889364 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889586 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889592 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889595 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889598 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889601 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889603 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889606 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889609 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889612 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889614 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889622 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889625 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889628 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889631 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:11.898103 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889633 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889636 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889638 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889643 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889646 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889648 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889651 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889654 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889660 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889663 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889665 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889668 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889670 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889673 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889675 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889679 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889683 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889691 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889695 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:11.899072 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889698 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889701 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889704 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889707 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889710 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889712 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889715 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889718 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889720 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889723 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889726 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889728 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889731 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889733 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889736 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889738 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889742 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889745 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889747 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889750 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:11.900027 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889752 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889755 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889758 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889760 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889763 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889765 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889768 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889770 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889773 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889775 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889778 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889788 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889791 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889794 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889796 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889799 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889801 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889804 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889807 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:11.900918 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889809 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889812 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889814 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889817 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889819 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889822 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889825 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889827 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889830 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889834 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889836 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889839 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889842 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889844 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889847 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889850 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889852 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889855 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889859 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:11.901759 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889862 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:11.902279 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889865 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:11.902279 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889867 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:11.902279 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889870 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:11.902279 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889873 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:11.902279 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889876 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:11.902279 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889884 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:11.902279 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889887 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:11.902279 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889890 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:11.902279 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889893 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:11.902279 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889895 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:11.902279 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889898 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:11.902279 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889900 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:11.902279 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889903 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:11.902279 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889906 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:11.902279 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.889933 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:51:11.902279 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889984 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889988 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889991 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889993 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889996 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.889999 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890001 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890005 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890009 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890013 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890017 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890020 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890022 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890025 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890027 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890030 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890033 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890035 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890038 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:11.902922 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890040 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890043 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890047 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890050 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890059 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890062 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890064 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890067 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890069 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890072 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890074 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890077 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890088 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890091 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890093 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890096 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890098 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890101 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890104 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890106 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:11.903708 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890110 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890113 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890116 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890119 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890122 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890124 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890127 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890130 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890133 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890135 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890138 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890140 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890143 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890145 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890151 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890155 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890158 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890168 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890172 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:11.904389 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890175 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890177 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890181 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890184 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890186 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890189 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890192 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890195 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890198 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890200 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890203 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890205 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890208 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890212 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890214 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890217 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890219 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890222 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890224 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890227 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:11.904952 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890230 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890232 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890235 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890237 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890240 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890242 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890245 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890249 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890251 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890254 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890257 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890266 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890269 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890272 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890274 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890277 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890279 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890282 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890285 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:11.905513 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890287 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890290 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890292 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890295 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890297 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890300 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890304 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890307 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.890310 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.899602 2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.899627 2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.899661 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.899670 2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899764 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899773 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899778 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899784 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899789 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899795 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:11.906033 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899799 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899804 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899808 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899812 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899817 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899822 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899830 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899836 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899841 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899846 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899850 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899854 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899861 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899868 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899872 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899877 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899881 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899887 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899892 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:11.906594 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899896 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899901 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899905 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899910 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899915 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899919 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899923 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899928 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899932 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899936 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899941 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899945 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899950 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899955 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899959 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899964 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899968 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899974 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899978 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899982 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:11.907150 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899987 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899993 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.899997 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900002 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900006 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900010 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900014 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900019 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900023 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900028 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900032 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900036 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900040 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900044 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900048 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900053 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900057 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900061 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900065 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:11.907649 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900070 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900075 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900079 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900083 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900088 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900092 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900096 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900100 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900104 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900109 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900114 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900118 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900122 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900127 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900131 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900137 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900141 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900146 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900150 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:11.908140 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900154 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900159 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900163 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900167 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900171 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900176 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900180 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900184 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900188 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900193 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900198 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900202 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900207 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900211 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900215 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900219 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900224 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900228 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900232 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900236 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:11.908622 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900241 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:11.909104 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900245 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:11.909104 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900249 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:11.909104 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.900292 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:51:11.909104 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900540 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:11.909104 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900548 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:11.909104 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900553 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:11.909104 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900558 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:11.909104 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900562 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:11.909104 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900567 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:11.909104 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900572 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:11.909104 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900576 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:11.909104 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900581 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:11.909104 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900585 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:11.909104 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900590 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:11.909104 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900594 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:11.909104 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900598 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900603 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900607 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900611 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900615 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900620 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900625 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900629 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900634 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900638 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900643 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900647 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900651 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900655 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900660 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900664 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900669 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900673 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900677 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:11.909574 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900681 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900685 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900689 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900693 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900698 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900702 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900706 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900711 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900716 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900720 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900724 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900728 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900732 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900739 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900745 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900750 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900754 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900759 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900763 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900768 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:11.910036 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900772 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900776 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900781 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900785 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900790 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900795 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900799 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900803 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900807 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900811 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900816 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900820 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900824 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900828 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900832 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900837 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900842 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900846 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900851 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900855 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:11.910634 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900860 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900865 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900869 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900873 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900877 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900881 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900886 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900890 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900894 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900898 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900903 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900907 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900914 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900920 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900927 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900931 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900936 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900940 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900945 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:11.911097 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900949 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:11.911564 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900953 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:11.911564 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900957 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:11.911564 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900962 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:11.911564 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900966 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:11.911564 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900970 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:11.911564 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900974 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:11.911564 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900979 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:11.911564 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900984 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:11.911564 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900988 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:11.911564 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900992 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:11.911564 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.900996 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:11.911564 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901001 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:11.911564 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901005 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:11.911564 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901014 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:11.911564 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901018 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:11.911564 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.901060 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901114 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901121 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901126 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901131 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901135 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901139 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901144 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901148 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901152 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901157 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901161 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901168 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901175 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901180 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901186 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901190 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901194 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901200 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:51:11.911971 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901204 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901209 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901215 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901219 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901224 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901229 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901233 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901237 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901242 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901246 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901251 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901257 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901263 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901267 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901271 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901275 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901280 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901284 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901288 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901292 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:51:11.912411 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901296 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901301 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901305 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901309 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901313 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901317 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901321 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901325 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901329 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901334 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901338 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901343 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901347 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901352 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901356 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901360 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901364 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901369 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901374 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:51:11.912903 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901379 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901383 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901387 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901392 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901398 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901403 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901408 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901412 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901417 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901421 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901426 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901430 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901434 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901438 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901442 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901446 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901453 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901458 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901462 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901466 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:51:11.913348 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901471 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901477 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901483 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901504 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901509 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901513 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901517 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901522 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901526 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901530 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901534 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901539 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901543 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901547 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901551 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901555 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901561 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901566 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901570 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901575 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:51:11.913860 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901579 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:51:11.914353 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901583 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:51:11.914353 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901588 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:51:11.914353 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901593 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:51:11.914353 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901599 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:51:11.914353 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901603 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:51:11.914353 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901607 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:51:11.914353 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901612 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:51:11.914353 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:11.901618 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:51:11.914353 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.901717 2572 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 12:51:11.914353 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.904869 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 12:51:11.914353 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.905858 2572 server.go:1014] "Starting client certificate rotation" Oct 30 12:51:11.914353 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.905958 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:51:11.914353 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.906526 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:51:11.914353 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.908031 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:51:11.914353 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.909192 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:51:11.918304 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.918285 2572 log.go:25] "Validated CRI v1 runtime API" Oct 30 12:51:11.921907 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.921888 2572 log.go:25] "Validated CRI v1 image API" Oct 30 12:51:11.923225 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.923203 2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 12:51:11.927472 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.927449 2572 fs.go:135] Filesystem UUIDs: map[540a3b59-f35a-45d8-bac1-e36ed708e86c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9084f533-0f76-4389-9afc-3b50c9997d9b:/dev/nvme0n1p3] Oct 30 12:51:11.927563 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.927470 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 12:51:11.934415 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.934288 2572 manager.go:217] Machine: {Timestamp:2026-10-30 12:51:11.931924177 +0000 UTC m=+0.318878070 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110951 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21e98d4438dd25b752966da07f68c3 SystemUUID:ec21e98d-4438-dd25-b752-966da07f68c3 BootID:6a9c628e-4a2d-4cbe-a29f-63493b9a4b06 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:de:d3:48:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:de:d3:48:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:a5:86:c7:81:78 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 12:51:11.935004 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.934992 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 12:51:11.935103 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.935085 2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 12:51:11.937955 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.937924 2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 12:51:11.938142 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.937956 2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-103.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 12:51:11.938224 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.938146 2572 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 12:51:11.938224 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.938159 2572 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 12:51:11.938224 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.938184 2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:51:11.938224 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.938207 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:51:11.939377 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.939364 2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 12:51:11.943750 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.943735 2572 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 12:51:11.943896 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.943885 2572 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 12:51:11.944612 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.944601 2572 kubelet.go:493] "Attempting to sync node with API server" Oct 30 12:51:11.944665 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.944618 2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 12:51:11.944665 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.944633 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 12:51:11.944665 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.944647 2572 kubelet.go:405] "Adding apiserver pod source" Oct 30 12:51:11.944665 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.944659 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 12:51:11.945035 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.945022 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:51:11.947276 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.947254 2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 12:51:11.953415 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.953398 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:51:11.954086 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.954064 2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 12:51:11.954151 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.954112 2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 12:51:11.954303 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.954290 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 12:51:11.954368 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.954310 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 12:51:11.954368 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.954320 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 12:51:11.954368 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.954330 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 12:51:11.954368 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.954339 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 12:51:11.954368 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.954347 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 12:51:11.954765 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.954754 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 12:51:11.954813 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.954769 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 12:51:11.954813 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.954780 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 12:51:11.954813 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.954790 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 12:51:11.954813 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.954804 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 12:51:11.954959 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.954818 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 12:51:11.955624 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.955614 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 12:51:11.955672 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.955628 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 12:51:11.959440 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.959424 2572 server.go:1263] "Started kubelet" Oct 30 12:51:11.959563 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.959515 2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 12:51:11.959991 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.959926 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 12:51:11.960099 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.960013 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 12:51:11.960209 ip-10-0-138-103 systemd[1]: Started Kubernetes Kubelet. Oct 30 12:51:11.961294 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.961275 2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 12:51:11.963125 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.962617 2572 server.go:317] "Adding debug handlers to kubelet server" Oct 30 12:51:11.965459 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:11.965418 2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 12:51:11.969628 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.969611 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 12:51:11.970589 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.970571 2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 12:51:11.970697 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:11.970679 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:51:11.971228 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.971212 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 12:51:11.971310 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.971242 2572 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 12:51:11.971310 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.971267 2572 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 12:51:11.971433 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.971349 2572 reconstruct.go:97] "Volume reconstruction finished" Oct 30 12:51:11.971433 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.971377 2572 reconciler.go:29] "Reconciler: start to sync state" Oct 30 12:51:11.971433 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.971380 2572 factory.go:55] Registering systemd factory Oct 30 12:51:11.971433 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.971392 2572 factory.go:223] Registration of the systemd container factory successfully Oct 30 12:51:11.971647 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.971634 2572 factory.go:153] Registering CRI-O factory Oct 30 12:51:11.971687 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.971651 2572 factory.go:223] Registration of the crio container factory successfully Oct 30 12:51:11.971730 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.971704 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 12:51:11.971730 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.971725 2572 factory.go:103] Registering Raw factory Oct 30 12:51:11.971795 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.971739 2572 manager.go:1199] Started watching for new ooms in manager Oct 30 12:51:11.972778 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.972761 2572 manager.go:319] Starting recovery of all containers Oct 30 12:51:11.986610 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.986562 2572 manager.go:324] Recovery completed Oct 30 12:51:11.991391 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.991377 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:11.993558 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.993534 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:11.993636 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.993572 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:11.993636 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.993584 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:11.994320 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.994307 2572 cpu_manager.go:226] "Starting" policy="none" Oct 30 12:51:11.994320 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.994317 2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 12:51:11.994424 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.994333 2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 12:51:11.994424 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.994385 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:11.995719 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.995698 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:11.995797 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.995730 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:11.995797 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.995745 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:11.996025 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.996001 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-103.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 12:51:11.996025 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:11.996014 2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-103.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 12:51:11.996617 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.996604 2572 policy_none.go:50] "Start" Oct 30 12:51:11.996654 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.996621 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 12:51:11.996654 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.996635 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 12:51:11.997020 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:11.996007 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-103.ec2.internal.18e34faae0dffb3c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-103.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-103.ec2.internal,},FirstTimestamp:2026-10-30 12:51:11.959399228 +0000 UTC m=+0.346353116,LastTimestamp:2026-10-30 12:51:11.959399228 +0000 UTC m=+0.346353116,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-103.ec2.internal,}" Oct 30 12:51:11.998569 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:11.998556 2572 policy_none.go:44] "Start" Oct 30 12:51:12.000099 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.000077 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 12:51:12.007379 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.007353 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dhwxm" Oct 30 12:51:12.009914 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.009888 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-103.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 12:51:12.010094 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.010060 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 12:51:12.010208 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.010191 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 12:51:12.020709 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.020684 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dhwxm" Oct 30 12:51:12.035230 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.035187 2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 12:51:12.039626 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.039611 2572 manager.go:341] "Starting Device Plugin manager" Oct 30 12:51:12.039721 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.039646 2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 12:51:12.039721 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.039657 2572 server.go:87] "Starting device plugin registration server" Oct 30 12:51:12.039961 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.039948 2572 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 12:51:12.040017 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.039963 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 12:51:12.040098 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.040086 2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 12:51:12.040179 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.040163 2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 12:51:12.040179 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.040175 2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 12:51:12.040803 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.040777 2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 12:51:12.040888 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.040822 2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:51:12.102970 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.102926 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 12:51:12.105007 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.104117 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 12:51:12.105354 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.105339 2572 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 12:51:12.105425 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.105375 2572 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 12:51:12.105425 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.105419 2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 12:51:12.123653 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.123617 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:12.141012 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.140941 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:12.143170 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.143148 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:12.143287 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.143181 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:12.143287 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.143196 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:12.143287 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.143225 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-103.ec2.internal" Oct 30 12:51:12.164680 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.164507 2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-103.ec2.internal" Oct 30 12:51:12.164788 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.164686 2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-103.ec2.internal\": node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:51:12.206074 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.206025 2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal"] Oct 30 12:51:12.206230 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.206113 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:12.207191 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.207175 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:12.207265 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.207216 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:12.207265 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.207228 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:12.209546 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.209533 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:12.209674 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.209660 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:51:12.209712 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.209689 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:12.210385 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.210368 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:12.210385 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.210381 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:12.210541 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.210395 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:12.210541 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.210403 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:12.210541 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.210405 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:12.210541 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.210413 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:12.212644 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.212627 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" Oct 30 12:51:12.212730 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.212653 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:12.213501 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.213476 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:12.213569 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.213514 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:12.213569 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.213551 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:12.216721 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.216704 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:51:12.243366 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.243342 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-103.ec2.internal\" not found" node="ip-10-0-138-103.ec2.internal" Oct 30 12:51:12.246041 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.246022 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-103.ec2.internal\" not found" node="ip-10-0-138-103.ec2.internal" Oct 30 12:51:12.317280 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.317250 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:51:12.373208 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.373174 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4c73d6ca462e65d2a8ead7238da805c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal\" (UID: \"b4c73d6ca462e65d2a8ead7238da805c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:51:12.373359 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.373226 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4c73d6ca462e65d2a8ead7238da805c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal\" (UID: \"b4c73d6ca462e65d2a8ead7238da805c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:51:12.373359 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.373259 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4db7be0b2dbb4b62fd854fb3bbef0e8f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-103.ec2.internal\" (UID: \"4db7be0b2dbb4b62fd854fb3bbef0e8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" Oct 30 12:51:12.417984 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.417907 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:51:12.473587 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.473556 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4c73d6ca462e65d2a8ead7238da805c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal\" (UID: \"b4c73d6ca462e65d2a8ead7238da805c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:51:12.473657 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.473592 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4c73d6ca462e65d2a8ead7238da805c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal\" (UID: \"b4c73d6ca462e65d2a8ead7238da805c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:51:12.473657 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.473637 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4db7be0b2dbb4b62fd854fb3bbef0e8f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-103.ec2.internal\" (UID: \"4db7be0b2dbb4b62fd854fb3bbef0e8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" Oct 30 12:51:12.473731 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.473672 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4db7be0b2dbb4b62fd854fb3bbef0e8f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-103.ec2.internal\" (UID: \"4db7be0b2dbb4b62fd854fb3bbef0e8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" Oct 30 12:51:12.473731 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.473675 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4c73d6ca462e65d2a8ead7238da805c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal\" (UID: \"b4c73d6ca462e65d2a8ead7238da805c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:51:12.473731 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.473724 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4c73d6ca462e65d2a8ead7238da805c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal\" (UID: \"b4c73d6ca462e65d2a8ead7238da805c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:51:12.518751 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.518715 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:51:12.547567 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.547538 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:51:12.550715 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.550693 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" Oct 30 12:51:12.619584 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.619552 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:51:12.721525 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.721440 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:51:12.822589 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.822542 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:51:12.906804 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.906762 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 12:51:12.907364 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:12.906948 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 12:51:12.923331 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:12.923287 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:51:13.023652 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:13.023615 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:51:13.026818 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.026778 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:46:12 +0000 UTC" deadline="2028-06-13 12:55:20.473406352 +0000 UTC" Oct 30 12:51:13.026818 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.026810 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14208h4m7.446599709s" Oct 30 12:51:13.124073 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:13.124043 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:51:13.148471 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:13.148430 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c73d6ca462e65d2a8ead7238da805c.slice/crio-98c3ddec071d956e18ff654cccc97b8264694b1c702c2c161017e23ea91bfd27 WatchSource:0}: Error finding container 98c3ddec071d956e18ff654cccc97b8264694b1c702c2c161017e23ea91bfd27: Status 404 returned error can't find the container with id 98c3ddec071d956e18ff654cccc97b8264694b1c702c2c161017e23ea91bfd27 Oct 30 12:51:13.148987 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:13.148972 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db7be0b2dbb4b62fd854fb3bbef0e8f.slice/crio-87ba1850a1899bb73ae42d0233d441fb994378aeee19d0c4cec52bad58abf701 WatchSource:0}: Error finding container 87ba1850a1899bb73ae42d0233d441fb994378aeee19d0c4cec52bad58abf701: Status 404 returned error can't find the container with id 87ba1850a1899bb73ae42d0233d441fb994378aeee19d0c4cec52bad58abf701 Oct 30 12:51:13.155718 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.155693 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:51:13.170312 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.170288 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:51:13.171747 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.171722 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:51:13.171845 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.171762 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:51:13.171845 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.171778 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:51:13.225153 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:13.225113 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:51:13.241422 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.241387 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:13.328135 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:13.328059 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:51:13.425465 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.425430 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:13.428798 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:13.428778 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:51:13.528904 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:13.528867 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:51:13.539061 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.539023 2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:13.571473 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.571436 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:51:13.606006 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.605922 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:51:13.606827 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.606800 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" Oct 30 12:51:13.630936 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.630894 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:51:13.853841 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.853802 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:51:13.945614 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.945525 2572 apiserver.go:52] "Watching apiserver" Oct 30 12:51:13.971380 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.971345 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 12:51:13.974089 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.974057 2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 12:51:13.974491 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.974463 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-bm8bc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal","openshift-multus/multus-4tbwx","openshift-network-diagnostics/network-check-target-dgnrj","openshift-ovn-kubernetes/ovnkube-node-jz7rd","openshift-cluster-node-tuning-operator/tuned-bsmf4","openshift-dns/node-resolver-qldmf","openshift-multus/multus-additional-cni-plugins-4mmzs","openshift-multus/network-metrics-daemon-b68v5","openshift-network-operator/iptables-alerter-r9nqr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k","kube-system/global-pull-secret-syncer-2xg75","kube-system/konnectivity-agent-zchsd","kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal"] Oct 30 12:51:13.977119 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.977088 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qldmf" Oct 30 12:51:13.979847 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.979477 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zchsd" Oct 30 12:51:13.982317 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.982295 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4tbwx" Oct 30 12:51:13.984702 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.984680 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:13.984810 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:13.984753 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:13.987067 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.987042 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:13.990978 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.989835 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:13.993033 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.992741 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:13.993033 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:13.992819 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:13.995697 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.995671 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:51:13.998632 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.998610 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:13.999307 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.999287 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hfgh7\"" Oct 30 12:51:13.999397 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.999375 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 12:51:13.999539 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.999499 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fv4k6\"" Oct 30 12:51:13.999772 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:13.999755 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9fz8p\"" Oct 30 12:51:14.001488 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.001465 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:51:14.002227 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.002205 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 12:51:14.002327 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.002228 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 12:51:14.002327 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.002286 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 12:51:14.002458 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.002373 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 12:51:14.002458 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.002383 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 12:51:14.002458 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.002431 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 12:51:14.002458 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.002444 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 12:51:14.003960 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.003940 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.006466 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.006439 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:14.006597 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:14.006518 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:14.010239 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.010218 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d8ng4\"" Oct 30 12:51:14.041510 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.041477 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 12:51:14.041710 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.041686 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kvkws" Oct 30 12:51:14.055554 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.055487 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kvkws" Oct 30 12:51:14.072636 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.072608 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 12:51:14.085062 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.084959 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-slash\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.085062 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085004 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-ovn\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.085062 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085037 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-log-socket\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.085359 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085111 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-kubernetes\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.085359 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085144 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-var-lib-kubelet\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.085359 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085222 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-hostroot\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.085359 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085282 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-run-netns\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.085359 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085312 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-node-log\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.085359 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085339 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-sys\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.085707 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085398 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-cni-binary-copy\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.085707 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085431 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-env-overrides\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.085707 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085463 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovn-node-metrics-cert\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.085707 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085545 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysctl-d\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.085707 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085623 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-host\") pod \"node-ca-bm8bc\" (UID: \"8bc2e7c1-04ef-439a-90e5-57c4c200c62f\") " pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:51:14.085707 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085641 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-system-cni-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.085707 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085657 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-cni-multus\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.086020 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085704 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6ctl\" (UniqueName: \"kubernetes.io/projected/050eebb4-7228-4189-b9ec-5bac3d7ed58e-kube-api-access-x6ctl\") pod \"network-metrics-daemon-b68v5\" (UID: \"050eebb4-7228-4189-b9ec-5bac3d7ed58e\") " pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:14.086020 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085771 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysctl-conf\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.086020 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085809 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/741107a5-c4db-4ba9-929c-58952377929f-agent-certs\") pod \"konnectivity-agent-zchsd\" (UID: \"741107a5-c4db-4ba9-929c-58952377929f\") " pod="kube-system/konnectivity-agent-zchsd" Oct 30 12:51:14.086020 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085856 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-cni-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.086020 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085902 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-system-cni-dir\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.086020 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085934 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-cnibin\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.086020 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085961 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-host\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.086020 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.085982 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-tmp\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.086020 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086008 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83545835-dd10-4f8b-8015-25c579bdac3a-host-slash\") pod \"iptables-alerter-r9nqr\" (UID: \"83545835-dd10-4f8b-8015-25c579bdac3a\") " pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:51:14.086323 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086037 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-registration-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.086323 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086100 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-systemd\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.086323 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086117 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-var-lib-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.086323 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086133 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-cni-bin\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.086323 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086148 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovnkube-script-lib\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.086323 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086165 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-serviceca\") pod \"node-ca-bm8bc\" (UID: \"8bc2e7c1-04ef-439a-90e5-57c4c200c62f\") " pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:51:14.086323 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086197 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-socket-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.086323 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086214 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gqh8\" (UniqueName: \"kubernetes.io/projected/109981ff-df1e-4279-bc7a-1e2b09f30ad1-kube-api-access-2gqh8\") pod \"node-resolver-qldmf\" (UID: \"109981ff-df1e-4279-bc7a-1e2b09f30ad1\") " pod="openshift-dns/node-resolver-qldmf" Oct 30 12:51:14.086323 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086230 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:14.086323 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086317 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/109981ff-df1e-4279-bc7a-1e2b09f30ad1-hosts-file\") pod \"node-resolver-qldmf\" (UID: \"109981ff-df1e-4279-bc7a-1e2b09f30ad1\") " pod="openshift-dns/node-resolver-qldmf" Oct 30 12:51:14.086725 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086340 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv4qf\" (UniqueName: \"kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf\") pod \"network-check-target-dgnrj\" (UID: \"2915194d-a9f7-460c-aee2-af5262da4af4\") " pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:14.086725 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086357 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-run-ovn-kubernetes\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.086725 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086386 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-tuned\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.086725 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086403 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-kubelet-config\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:14.086725 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086417 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-conf-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.086725 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086433 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-daemon-config\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.086725 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086449 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-modprobe-d\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.086725 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086470 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysconfig\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.086725 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086496 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dpcr\" (UniqueName: \"kubernetes.io/projected/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-kube-api-access-4dpcr\") pod \"node-ca-bm8bc\" (UID: \"8bc2e7c1-04ef-439a-90e5-57c4c200c62f\") " pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:51:14.086725 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086519 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-etc-selinux\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.086725 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086535 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-cnibin\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.086725 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086575 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs\") pod \"network-metrics-daemon-b68v5\" (UID: \"050eebb4-7228-4189-b9ec-5bac3d7ed58e\") " pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:14.086725 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086592 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.086725 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086610 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-kubelet-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.086725 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086627 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6fp9\" (UniqueName: \"kubernetes.io/projected/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-kube-api-access-x6fp9\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.086725 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086715 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.087306 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086736 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvg8b\" (UniqueName: \"kubernetes.io/projected/9b663748-9129-4d2c-b900-c5ddde8b4b63-kube-api-access-dvg8b\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.087306 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086761 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-lib-modules\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.087306 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086776 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5q2d\" (UniqueName: \"kubernetes.io/projected/83545835-dd10-4f8b-8015-25c579bdac3a-kube-api-access-w5q2d\") pod \"iptables-alerter-r9nqr\" (UID: \"83545835-dd10-4f8b-8015-25c579bdac3a\") " pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:51:14.087306 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086793 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/741107a5-c4db-4ba9-929c-58952377929f-konnectivity-ca\") pod \"konnectivity-agent-zchsd\" (UID: \"741107a5-c4db-4ba9-929c-58952377929f\") " pod="kube-system/konnectivity-agent-zchsd" Oct 30 12:51:14.087306 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086809 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-k8s-cni-cncf-io\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.087306 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086833 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-kubelet\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.087306 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086852 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-cni-netd\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.087306 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086872 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-dbus\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:14.087306 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086889 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.087306 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086906 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c62x8\" (UniqueName: \"kubernetes.io/projected/fce77b3b-df3e-4d04-945c-68dc11edc61f-kube-api-access-c62x8\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.087306 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086921 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/109981ff-df1e-4279-bc7a-1e2b09f30ad1-tmp-dir\") pod \"node-resolver-qldmf\" (UID: \"109981ff-df1e-4279-bc7a-1e2b09f30ad1\") " pod="openshift-dns/node-resolver-qldmf" Oct 30 12:51:14.087306 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086939 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-sys-fs\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.087306 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086955 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-etc-kubernetes\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.087306 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.086981 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.087306 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.087000 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-kubelet\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.087306 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.087016 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-run\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.087918 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.087037 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kwws\" (UniqueName: \"kubernetes.io/projected/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-kube-api-access-5kwws\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.087918 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.087054 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83545835-dd10-4f8b-8015-25c579bdac3a-iptables-alerter-script\") pod \"iptables-alerter-r9nqr\" (UID: \"83545835-dd10-4f8b-8015-25c579bdac3a\") " pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:51:14.087918 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.087080 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-netns\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.087918 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.087096 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-cni-bin\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.087918 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.087119 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-multus-certs\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.087918 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.087138 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-cni-binary-copy\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.087918 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.087154 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-etc-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.087918 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.087170 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovnkube-config\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.087918 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.087189 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-systemd\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.087918 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.087215 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-device-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.087918 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.087245 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd6nk\" (UniqueName: \"kubernetes.io/projected/404bb84d-f044-488e-a221-389f71e61181-kube-api-access-hd6nk\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.087918 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.087264 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.087918 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.087281 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-systemd-units\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.087918 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.087426 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-socket-dir-parent\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.108313 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.107864 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 12:51:14.108313 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.107919 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 12:51:14.108313 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.108060 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-588rd\"" Oct 30 12:51:14.108313 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.108073 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:14.108313 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.108099 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:51:14.108313 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.108161 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 12:51:14.108313 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.108198 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:14.108313 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.108216 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 12:51:14.108313 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.108247 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 12:51:14.108313 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.108256 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 12:51:14.108313 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.108267 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 12:51:14.108313 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.108287 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 12:51:14.109055 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.108581 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 12:51:14.110098 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.109667 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 12:51:14.110098 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.109821 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g6csn\"" Oct 30 12:51:14.110098 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.109973 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 12:51:14.121787 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.121686 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nklv7\"" Oct 30 12:51:14.121787 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.121775 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 12:51:14.122241 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.122217 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pdl9w\"" Oct 30 12:51:14.122336 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.122324 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 12:51:14.122386 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.122362 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-78z4l\"" Oct 30 12:51:14.122425 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.122395 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 12:51:14.122470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.122420 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 12:51:14.122731 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.122672 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 12:51:14.124096 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.124039 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" event={"ID":"b4c73d6ca462e65d2a8ead7238da805c","Type":"ContainerStarted","Data":"98c3ddec071d956e18ff654cccc97b8264694b1c702c2c161017e23ea91bfd27"} Oct 30 12:51:14.125555 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.125505 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" event={"ID":"4db7be0b2dbb4b62fd854fb3bbef0e8f","Type":"ContainerStarted","Data":"87ba1850a1899bb73ae42d0233d441fb994378aeee19d0c4cec52bad58abf701"} Oct 30 12:51:14.188094 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188054 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.188094 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188107 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-kubelet\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.188357 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188137 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-run\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.188357 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188176 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kwws\" (UniqueName: \"kubernetes.io/projected/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-kube-api-access-5kwws\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.188357 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188207 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83545835-dd10-4f8b-8015-25c579bdac3a-iptables-alerter-script\") pod \"iptables-alerter-r9nqr\" (UID: \"83545835-dd10-4f8b-8015-25c579bdac3a\") " pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:51:14.188357 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188233 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-netns\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.188357 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188258 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-cni-bin\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.188357 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188283 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-multus-certs\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.188357 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188293 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-kubelet\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.188357 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188310 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-cni-binary-copy\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.188357 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188335 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-etc-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.188357 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188333 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-run\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.188357 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188355 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-netns\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.189004 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188400 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-multus-certs\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.189004 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188600 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovnkube-config\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.189004 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188735 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-cni-bin\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.189004 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188769 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-systemd\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.189004 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188796 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-device-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.189004 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188818 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd6nk\" (UniqueName: \"kubernetes.io/projected/404bb84d-f044-488e-a221-389f71e61181-kube-api-access-hd6nk\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.189004 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188872 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.189004 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188883 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-cni-binary-copy\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.189004 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188886 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-device-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.189004 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188934 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.189004 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188942 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-etc-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.189004 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188948 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83545835-dd10-4f8b-8015-25c579bdac3a-iptables-alerter-script\") pod \"iptables-alerter-r9nqr\" (UID: \"83545835-dd10-4f8b-8015-25c579bdac3a\") " pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:51:14.189004 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.188969 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-systemd-units\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189018 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-systemd\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189084 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-socket-dir-parent\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189120 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-socket-dir-parent\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189134 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-slash\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189161 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-ovn\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189178 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-systemd-units\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189215 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-slash\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189220 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-ovn\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189225 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189275 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-log-socket\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189295 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-kubernetes\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189310 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-var-lib-kubelet\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189325 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-hostroot\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189337 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-log-socket\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189350 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-run-netns\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189390 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-node-log\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189408 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-hostroot\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.189655 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189464 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-node-log\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189472 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-kubernetes\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189421 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-sys\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189510 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-var-lib-kubelet\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189526 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-sys\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189527 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovnkube-config\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189533 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-cni-binary-copy\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189559 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-run-netns\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189596 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-env-overrides\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189625 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovn-node-metrics-cert\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189650 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysctl-d\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189695 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-host\") pod \"node-ca-bm8bc\" (UID: \"8bc2e7c1-04ef-439a-90e5-57c4c200c62f\") " pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189723 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-system-cni-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189750 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-cni-multus\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189762 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-host\") pod \"node-ca-bm8bc\" (UID: \"8bc2e7c1-04ef-439a-90e5-57c4c200c62f\") " pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189776 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ctl\" (UniqueName: \"kubernetes.io/projected/050eebb4-7228-4189-b9ec-5bac3d7ed58e-kube-api-access-x6ctl\") pod \"network-metrics-daemon-b68v5\" (UID: \"050eebb4-7228-4189-b9ec-5bac3d7ed58e\") " pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189791 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysctl-d\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189827 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-system-cni-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.190470 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189842 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-cni-multus\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysctl-conf\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189892 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/741107a5-c4db-4ba9-929c-58952377929f-agent-certs\") pod \"konnectivity-agent-zchsd\" (UID: \"741107a5-c4db-4ba9-929c-58952377929f\") " pod="kube-system/konnectivity-agent-zchsd" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189918 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-cni-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189952 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-system-cni-dir\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189979 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-cnibin\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.189989 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysctl-conf\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190013 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-host\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190037 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-tmp\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190054 2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190068 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83545835-dd10-4f8b-8015-25c579bdac3a-host-slash\") pod \"iptables-alerter-r9nqr\" (UID: \"83545835-dd10-4f8b-8015-25c579bdac3a\") " pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190070 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-cnibin\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190083 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-env-overrides\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190091 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-cni-binary-copy\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190108 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-registration-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190112 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-cni-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190060 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-system-cni-dir\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190110 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-host\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.191139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190112 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83545835-dd10-4f8b-8015-25c579bdac3a-host-slash\") pod \"iptables-alerter-r9nqr\" (UID: \"83545835-dd10-4f8b-8015-25c579bdac3a\") " pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:51:14.191919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190155 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-systemd\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.191919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190174 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-systemd\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.191919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190158 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-registration-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.191919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190184 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-var-lib-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.191919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190229 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-var-lib-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.191919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190242 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-cni-bin\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.191919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190272 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovnkube-script-lib\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.191919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190297 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-serviceca\") pod \"node-ca-bm8bc\" (UID: \"8bc2e7c1-04ef-439a-90e5-57c4c200c62f\") " pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:51:14.191919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190323 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-socket-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.191919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190330 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-cni-bin\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.191919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190348 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gqh8\" (UniqueName: \"kubernetes.io/projected/109981ff-df1e-4279-bc7a-1e2b09f30ad1-kube-api-access-2gqh8\") pod \"node-resolver-qldmf\" (UID: \"109981ff-df1e-4279-bc7a-1e2b09f30ad1\") " pod="openshift-dns/node-resolver-qldmf" Oct 30 12:51:14.191919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190379 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:14.191919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190410 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/109981ff-df1e-4279-bc7a-1e2b09f30ad1-hosts-file\") pod \"node-resolver-qldmf\" (UID: \"109981ff-df1e-4279-bc7a-1e2b09f30ad1\") " pod="openshift-dns/node-resolver-qldmf" Oct 30 12:51:14.191919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190442 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4qf\" (UniqueName: \"kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf\") pod \"network-check-target-dgnrj\" (UID: \"2915194d-a9f7-460c-aee2-af5262da4af4\") " pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:14.191919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190462 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-socket-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.191919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190467 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-run-ovn-kubernetes\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.191919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190523 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-run-ovn-kubernetes\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.192588 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190544 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-tuned\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.192588 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190573 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-kubelet-config\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:14.192588 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190601 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-conf-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.192588 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190627 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-daemon-config\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.192588 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190653 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-modprobe-d\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.192588 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190677 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysconfig\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.192588 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190704 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dpcr\" (UniqueName: \"kubernetes.io/projected/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-kube-api-access-4dpcr\") pod \"node-ca-bm8bc\" (UID: \"8bc2e7c1-04ef-439a-90e5-57c4c200c62f\") " pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:51:14.192588 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190732 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-etc-selinux\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.192588 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190742 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-serviceca\") pod \"node-ca-bm8bc\" (UID: \"8bc2e7c1-04ef-439a-90e5-57c4c200c62f\") " pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:51:14.192588 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190757 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-cnibin\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.192588 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190783 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs\") pod \"network-metrics-daemon-b68v5\" (UID: \"050eebb4-7228-4189-b9ec-5bac3d7ed58e\") " pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:14.192588 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190808 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.192588 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190834 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-kubelet-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.192588 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190861 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fp9\" (UniqueName: \"kubernetes.io/projected/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-kube-api-access-x6fp9\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.192588 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:14.190873 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:14.192588 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190888 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.192588 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190917 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvg8b\" (UniqueName: \"kubernetes.io/projected/9b663748-9129-4d2c-b900-c5ddde8b4b63-kube-api-access-dvg8b\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.193220 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:14.190938 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret podName:0b497ad2-5bd4-4b7b-881d-d94c15fda49c nodeName:}" failed. No retries permitted until 2026-10-30 12:51:14.690918639 +0000 UTC m=+3.077872536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret") pod "global-pull-secret-syncer-2xg75" (UID: "0b497ad2-5bd4-4b7b-881d-d94c15fda49c") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:14.193220 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190967 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-lib-modules\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.193220 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5q2d\" (UniqueName: \"kubernetes.io/projected/83545835-dd10-4f8b-8015-25c579bdac3a-kube-api-access-w5q2d\") pod \"iptables-alerter-r9nqr\" (UID: \"83545835-dd10-4f8b-8015-25c579bdac3a\") " pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:51:14.193220 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191018 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/741107a5-c4db-4ba9-929c-58952377929f-konnectivity-ca\") pod \"konnectivity-agent-zchsd\" (UID: \"741107a5-c4db-4ba9-929c-58952377929f\") " pod="kube-system/konnectivity-agent-zchsd" Oct 30 12:51:14.193220 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191040 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-k8s-cni-cncf-io\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.193220 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191062 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-kubelet\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.193220 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191085 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-cni-netd\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.193220 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191108 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-dbus\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:14.193220 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191130 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.193220 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191140 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-etc-selinux\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.193220 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191154 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c62x8\" (UniqueName: \"kubernetes.io/projected/fce77b3b-df3e-4d04-945c-68dc11edc61f-kube-api-access-c62x8\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.193220 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191182 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/109981ff-df1e-4279-bc7a-1e2b09f30ad1-tmp-dir\") pod \"node-resolver-qldmf\" (UID: \"109981ff-df1e-4279-bc7a-1e2b09f30ad1\") " pod="openshift-dns/node-resolver-qldmf" Oct 30 12:51:14.193220 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191186 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-cnibin\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.193220 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191206 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-sys-fs\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.193220 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191230 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-etc-kubernetes\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.193220 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191241 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-kubelet\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.193220 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:14.191247 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:14.193827 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.190810 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/109981ff-df1e-4279-bc7a-1e2b09f30ad1-hosts-file\") pod \"node-resolver-qldmf\" (UID: \"109981ff-df1e-4279-bc7a-1e2b09f30ad1\") " pod="openshift-dns/node-resolver-qldmf" Oct 30 12:51:14.193827 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191316 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.193827 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191329 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-cni-netd\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.193827 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:14.191337 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs podName:050eebb4-7228-4189-b9ec-5bac3d7ed58e nodeName:}" failed. No retries permitted until 2026-10-30 12:51:14.6913238 +0000 UTC m=+3.078277711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs") pod "network-metrics-daemon-b68v5" (UID: "050eebb4-7228-4189-b9ec-5bac3d7ed58e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:14.193827 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191355 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-sys-fs\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.193827 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191402 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-dbus\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:14.193827 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191437 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-etc-kubernetes\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.193827 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191441 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysconfig\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.193827 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191437 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-lib-modules\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.193827 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191469 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-kubelet-config\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:14.193827 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191471 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.193827 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191668 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-modprobe-d\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.193827 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191706 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-conf-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.193827 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191742 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-k8s-cni-cncf-io\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.193827 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191789 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovnkube-script-lib\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.193827 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191808 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/109981ff-df1e-4279-bc7a-1e2b09f30ad1-tmp-dir\") pod \"node-resolver-qldmf\" (UID: \"109981ff-df1e-4279-bc7a-1e2b09f30ad1\") " pod="openshift-dns/node-resolver-qldmf" Oct 30 12:51:14.193827 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.191838 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-kubelet-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.194403 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.192080 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-daemon-config\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.194403 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.192247 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.194403 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.192349 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/741107a5-c4db-4ba9-929c-58952377929f-konnectivity-ca\") pod \"konnectivity-agent-zchsd\" (UID: \"741107a5-c4db-4ba9-929c-58952377929f\") " pod="kube-system/konnectivity-agent-zchsd" Oct 30 12:51:14.194635 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.194607 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-tuned\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.195128 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.195098 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-tmp\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.195315 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.195292 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/741107a5-c4db-4ba9-929c-58952377929f-agent-certs\") pod \"konnectivity-agent-zchsd\" (UID: \"741107a5-c4db-4ba9-929c-58952377929f\") " pod="kube-system/konnectivity-agent-zchsd" Oct 30 12:51:14.196939 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.196878 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovn-node-metrics-cert\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.213709 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.213683 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd6nk\" (UniqueName: \"kubernetes.io/projected/404bb84d-f044-488e-a221-389f71e61181-kube-api-access-hd6nk\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.214159 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.214141 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kwws\" (UniqueName: \"kubernetes.io/projected/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-kube-api-access-5kwws\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.225183 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.225149 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gqh8\" (UniqueName: \"kubernetes.io/projected/109981ff-df1e-4279-bc7a-1e2b09f30ad1-kube-api-access-2gqh8\") pod \"node-resolver-qldmf\" (UID: \"109981ff-df1e-4279-bc7a-1e2b09f30ad1\") " pod="openshift-dns/node-resolver-qldmf" Oct 30 12:51:14.225399 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.225375 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6ctl\" (UniqueName: \"kubernetes.io/projected/050eebb4-7228-4189-b9ec-5bac3d7ed58e-kube-api-access-x6ctl\") pod \"network-metrics-daemon-b68v5\" (UID: \"050eebb4-7228-4189-b9ec-5bac3d7ed58e\") " pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:14.227511 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.227482 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvg8b\" (UniqueName: \"kubernetes.io/projected/9b663748-9129-4d2c-b900-c5ddde8b4b63-kube-api-access-dvg8b\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.228134 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.228109 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dpcr\" (UniqueName: \"kubernetes.io/projected/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-kube-api-access-4dpcr\") pod \"node-ca-bm8bc\" (UID: \"8bc2e7c1-04ef-439a-90e5-57c4c200c62f\") " pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:51:14.233380 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:14.233355 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:14.233528 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:14.233386 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:14.233528 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:14.233401 2572 projected.go:196] Error preparing data for projected volume kube-api-access-fv4qf for pod openshift-network-diagnostics/network-check-target-dgnrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:14.233528 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:14.233475 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf podName:2915194d-a9f7-460c-aee2-af5262da4af4 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:14.733451121 +0000 UTC m=+3.120405018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fv4qf" (UniqueName: "kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf") pod "network-check-target-dgnrj" (UID: "2915194d-a9f7-460c-aee2-af5262da4af4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:14.234949 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.234903 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6fp9\" (UniqueName: \"kubernetes.io/projected/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-kube-api-access-x6fp9\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.235054 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.235011 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c62x8\" (UniqueName: \"kubernetes.io/projected/fce77b3b-df3e-4d04-945c-68dc11edc61f-kube-api-access-c62x8\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.237836 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.237810 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5q2d\" (UniqueName: \"kubernetes.io/projected/83545835-dd10-4f8b-8015-25c579bdac3a-kube-api-access-w5q2d\") pod \"iptables-alerter-r9nqr\" (UID: \"83545835-dd10-4f8b-8015-25c579bdac3a\") " pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:51:14.295038 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.295000 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qldmf" Oct 30 12:51:14.302919 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.302896 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zchsd" Oct 30 12:51:14.307098 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:14.307064 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod109981ff_df1e_4279_bc7a_1e2b09f30ad1.slice/crio-ecf7cf31b72f03a1a62c9eb61c6584dd6751f35a49c9bf5e44d9b9433ee9f31f WatchSource:0}: Error finding container ecf7cf31b72f03a1a62c9eb61c6584dd6751f35a49c9bf5e44d9b9433ee9f31f: Status 404 returned error can't find the container with id ecf7cf31b72f03a1a62c9eb61c6584dd6751f35a49c9bf5e44d9b9433ee9f31f Oct 30 12:51:14.313501 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.313451 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4tbwx" Oct 30 12:51:14.314008 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:14.313973 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741107a5_c4db_4ba9_929c_58952377929f.slice/crio-e9f1e9c6bb635bc9663b6fad1d374d211b92838f8bcf96afa913ecde4e5147e0 WatchSource:0}: Error finding container e9f1e9c6bb635bc9663b6fad1d374d211b92838f8bcf96afa913ecde4e5147e0: Status 404 returned error can't find the container with id e9f1e9c6bb635bc9663b6fad1d374d211b92838f8bcf96afa913ecde4e5147e0 Oct 30 12:51:14.324748 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:14.324717 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b7ca3d_22f2_4465_81a7_3a2d4082a6e9.slice/crio-153896f3b4cb172c890a65850fdd9f5159745eac48318cf3426b69f348ecc283 WatchSource:0}: Error finding container 153896f3b4cb172c890a65850fdd9f5159745eac48318cf3426b69f348ecc283: Status 404 returned error can't find the container with id 153896f3b4cb172c890a65850fdd9f5159745eac48318cf3426b69f348ecc283 Oct 30 12:51:14.334932 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.334900 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:51:14.340646 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.340399 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:14.341926 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.341904 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:51:14.347273 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:14.347246 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe4fa10_9b21_4d40_ae05_13bc0cfb67dd.slice/crio-123ac7185c11535af0a99c44bf3d387108ce9268396f963347a5980a3e8cfc1e WatchSource:0}: Error finding container 123ac7185c11535af0a99c44bf3d387108ce9268396f963347a5980a3e8cfc1e: Status 404 returned error can't find the container with id 123ac7185c11535af0a99c44bf3d387108ce9268396f963347a5980a3e8cfc1e Oct 30 12:51:14.355811 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:14.355784 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce77b3b_df3e_4d04_945c_68dc11edc61f.slice/crio-11dbf715ab7856ce862a47a1fee18a01aa768b32f7005a999e8af99b8df53c82 WatchSource:0}: Error finding container 11dbf715ab7856ce862a47a1fee18a01aa768b32f7005a999e8af99b8df53c82: Status 404 returned error can't find the container with id 11dbf715ab7856ce862a47a1fee18a01aa768b32f7005a999e8af99b8df53c82 Oct 30 12:51:14.356429 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:14.356410 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc2e7c1_04ef_439a_90e5_57c4c200c62f.slice/crio-91404ee3622320376e4465fbd87101539743f97d8873143673c38ae1a64368fe WatchSource:0}: Error finding container 91404ee3622320376e4465fbd87101539743f97d8873143673c38ae1a64368fe: Status 404 returned error can't find the container with id 91404ee3622320376e4465fbd87101539743f97d8873143673c38ae1a64368fe Oct 30 12:51:14.362820 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.362800 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:51:14.364208 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.364184 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:51:14.372915 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.372890 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:51:14.376125 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:14.376085 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83545835_dd10_4f8b_8015_25c579bdac3a.slice/crio-4a3430d1e2186a2de5e9637a00a4c6d50862f1804bfb8fed6b6086c0a2cf528c WatchSource:0}: Error finding container 4a3430d1e2186a2de5e9637a00a4c6d50862f1804bfb8fed6b6086c0a2cf528c: Status 404 returned error can't find the container with id 4a3430d1e2186a2de5e9637a00a4c6d50862f1804bfb8fed6b6086c0a2cf528c Oct 30 12:51:14.386169 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:51:14.386115 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404bb84d_f044_488e_a221_389f71e61181.slice/crio-2c8c1de77cf6c4cbda32164f7696a993e56def4e4db1de3840cb5ab2f7663344 WatchSource:0}: Error finding container 2c8c1de77cf6c4cbda32164f7696a993e56def4e4db1de3840cb5ab2f7663344: Status 404 returned error can't find the container with id 2c8c1de77cf6c4cbda32164f7696a993e56def4e4db1de3840cb5ab2f7663344 Oct 30 12:51:14.694907 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.694874 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:14.695030 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.694939 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs\") pod \"network-metrics-daemon-b68v5\" (UID: \"050eebb4-7228-4189-b9ec-5bac3d7ed58e\") " pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:14.695634 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:14.695109 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:14.695634 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:14.695176 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs podName:050eebb4-7228-4189-b9ec-5bac3d7ed58e nodeName:}" failed. No retries permitted until 2026-10-30 12:51:15.695160164 +0000 UTC m=+4.082114054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs") pod "network-metrics-daemon-b68v5" (UID: "050eebb4-7228-4189-b9ec-5bac3d7ed58e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:14.695634 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:14.695553 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:14.695634 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:14.695616 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret podName:0b497ad2-5bd4-4b7b-881d-d94c15fda49c nodeName:}" failed. No retries permitted until 2026-10-30 12:51:15.695604268 +0000 UTC m=+4.082558144 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret") pod "global-pull-secret-syncer-2xg75" (UID: "0b497ad2-5bd4-4b7b-881d-d94c15fda49c") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:14.795973 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:14.795701 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4qf\" (UniqueName: \"kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf\") pod \"network-check-target-dgnrj\" (UID: \"2915194d-a9f7-460c-aee2-af5262da4af4\") " pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:14.796173 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:14.796153 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:14.796242 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:14.796182 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:14.796242 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:14.796197 2572 projected.go:196] Error preparing data for projected volume kube-api-access-fv4qf for pod openshift-network-diagnostics/network-check-target-dgnrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:14.796352 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:14.796259 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf podName:2915194d-a9f7-460c-aee2-af5262da4af4 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:15.796241171 +0000 UTC m=+4.183195051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fv4qf" (UniqueName: "kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf") pod "network-check-target-dgnrj" (UID: "2915194d-a9f7-460c-aee2-af5262da4af4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:15.058810 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:15.058740 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:46:14 +0000 UTC" deadline="2028-04-21 10:19:46.940682185 +0000 UTC" Oct 30 12:51:15.058810 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:15.058809 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12933h28m31.881877973s" Oct 30 12:51:15.136396 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:15.136299 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" event={"ID":"404bb84d-f044-488e-a221-389f71e61181","Type":"ContainerStarted","Data":"2c8c1de77cf6c4cbda32164f7696a993e56def4e4db1de3840cb5ab2f7663344"} Oct 30 12:51:15.138691 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:15.138609 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mmzs" event={"ID":"9b663748-9129-4d2c-b900-c5ddde8b4b63","Type":"ContainerStarted","Data":"29ccd7fa19dcd4c1dfac94a30a27d98cce2a48b7f3a8782fc46acb5e8f2994a8"} Oct 30 12:51:15.142625 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:15.142506 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" event={"ID":"fce77b3b-df3e-4d04-945c-68dc11edc61f","Type":"ContainerStarted","Data":"11dbf715ab7856ce862a47a1fee18a01aa768b32f7005a999e8af99b8df53c82"} Oct 30 12:51:15.166458 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:15.166413 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" event={"ID":"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd","Type":"ContainerStarted","Data":"123ac7185c11535af0a99c44bf3d387108ce9268396f963347a5980a3e8cfc1e"} Oct 30 12:51:15.174794 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:15.174756 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zchsd" event={"ID":"741107a5-c4db-4ba9-929c-58952377929f","Type":"ContainerStarted","Data":"e9f1e9c6bb635bc9663b6fad1d374d211b92838f8bcf96afa913ecde4e5147e0"} Oct 30 12:51:15.178172 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:15.178138 2572 generic.go:356] "Generic (PLEG): container finished" podID="b4c73d6ca462e65d2a8ead7238da805c" containerID="e30a112d7a41fdcd2f677bbb475da5cd078e5a53e71ebdc49e797038e2462858" exitCode=0 Oct 30 12:51:15.178319 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:15.178290 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" event={"ID":"b4c73d6ca462e65d2a8ead7238da805c","Type":"ContainerDied","Data":"e30a112d7a41fdcd2f677bbb475da5cd078e5a53e71ebdc49e797038e2462858"} Oct 30 12:51:15.183838 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:15.183292 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" event={"ID":"4db7be0b2dbb4b62fd854fb3bbef0e8f","Type":"ContainerStarted","Data":"7800f79398bd4aebfc552339785fe1a0fe724fd489d7b3a7c656b04ea57dbeec"} Oct 30 12:51:15.189312 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:15.189237 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r9nqr" event={"ID":"83545835-dd10-4f8b-8015-25c579bdac3a","Type":"ContainerStarted","Data":"4a3430d1e2186a2de5e9637a00a4c6d50862f1804bfb8fed6b6086c0a2cf528c"} Oct 30 12:51:15.191224 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:15.191143 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bm8bc" event={"ID":"8bc2e7c1-04ef-439a-90e5-57c4c200c62f","Type":"ContainerStarted","Data":"91404ee3622320376e4465fbd87101539743f97d8873143673c38ae1a64368fe"} Oct 30 12:51:15.197745 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:15.197655 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4tbwx" event={"ID":"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9","Type":"ContainerStarted","Data":"153896f3b4cb172c890a65850fdd9f5159745eac48318cf3426b69f348ecc283"} Oct 30 12:51:15.201934 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:15.201901 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qldmf" event={"ID":"109981ff-df1e-4279-bc7a-1e2b09f30ad1","Type":"ContainerStarted","Data":"ecf7cf31b72f03a1a62c9eb61c6584dd6751f35a49c9bf5e44d9b9433ee9f31f"} Oct 30 12:51:15.703104 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:15.703001 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:15.703104 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:15.703079 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs\") pod \"network-metrics-daemon-b68v5\" (UID: \"050eebb4-7228-4189-b9ec-5bac3d7ed58e\") " pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:15.703334 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:15.703263 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:15.703334 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:15.703331 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs podName:050eebb4-7228-4189-b9ec-5bac3d7ed58e nodeName:}" failed. No retries permitted until 2026-10-30 12:51:17.703309502 +0000 UTC m=+6.090263395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs") pod "network-metrics-daemon-b68v5" (UID: "050eebb4-7228-4189-b9ec-5bac3d7ed58e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:15.703774 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:15.703753 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:15.703879 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:15.703812 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret podName:0b497ad2-5bd4-4b7b-881d-d94c15fda49c nodeName:}" failed. No retries permitted until 2026-10-30 12:51:17.703795908 +0000 UTC m=+6.090749798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret") pod "global-pull-secret-syncer-2xg75" (UID: "0b497ad2-5bd4-4b7b-881d-d94c15fda49c") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:15.805118 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:15.804484 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4qf\" (UniqueName: \"kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf\") pod \"network-check-target-dgnrj\" (UID: \"2915194d-a9f7-460c-aee2-af5262da4af4\") " pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:15.805118 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:15.804648 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:15.805118 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:15.804677 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:15.805118 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:15.804692 2572 projected.go:196] Error preparing data for projected volume kube-api-access-fv4qf for pod openshift-network-diagnostics/network-check-target-dgnrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:15.805118 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:15.804763 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf podName:2915194d-a9f7-460c-aee2-af5262da4af4 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:17.804742254 +0000 UTC m=+6.191696145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fv4qf" (UniqueName: "kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf") pod "network-check-target-dgnrj" (UID: "2915194d-a9f7-460c-aee2-af5262da4af4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:16.060893 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:16.060707 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:46:14 +0000 UTC" deadline="2028-07-26 11:41:32.84747453 +0000 UTC" Oct 30 12:51:16.060893 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:16.060772 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15238h50m16.786707221s" Oct 30 12:51:16.120281 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:16.120252 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:16.120472 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:16.120373 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:16.121023 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:16.120999 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:16.121164 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:16.121143 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:16.121235 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:16.121222 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:16.121348 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:16.121328 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:16.208729 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:16.208693 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" event={"ID":"b4c73d6ca462e65d2a8ead7238da805c","Type":"ContainerStarted","Data":"268c99143ecbb1f541c174b80afc75a748af78999a9789cc006abe73c7d3d1c9"} Oct 30 12:51:16.298048 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:16.297177 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" podStartSLOduration=3.297156037 podStartE2EDuration="3.297156037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:15.261823037 +0000 UTC m=+3.648776940" watchObservedRunningTime="2026-10-30 12:51:16.297156037 +0000 UTC m=+4.684109941" Oct 30 12:51:16.298048 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:16.297365 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" podStartSLOduration=3.297356391 podStartE2EDuration="3.297356391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:16.296488517 +0000 UTC m=+4.683442419" watchObservedRunningTime="2026-10-30 12:51:16.297356391 +0000 UTC m=+4.684310292" Oct 30 12:51:17.723121 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:17.723003 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:17.723121 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:17.723077 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs\") pod \"network-metrics-daemon-b68v5\" (UID: \"050eebb4-7228-4189-b9ec-5bac3d7ed58e\") " pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:17.723665 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:17.723227 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:17.723665 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:17.723286 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs podName:050eebb4-7228-4189-b9ec-5bac3d7ed58e nodeName:}" failed. No retries permitted until 2026-10-30 12:51:21.723268506 +0000 UTC m=+10.110222398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs") pod "network-metrics-daemon-b68v5" (UID: "050eebb4-7228-4189-b9ec-5bac3d7ed58e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:17.723774 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:17.723703 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:17.723774 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:17.723756 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret podName:0b497ad2-5bd4-4b7b-881d-d94c15fda49c nodeName:}" failed. No retries permitted until 2026-10-30 12:51:21.723740364 +0000 UTC m=+10.110694254 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret") pod "global-pull-secret-syncer-2xg75" (UID: "0b497ad2-5bd4-4b7b-881d-d94c15fda49c") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:17.823624 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:17.823586 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4qf\" (UniqueName: \"kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf\") pod \"network-check-target-dgnrj\" (UID: \"2915194d-a9f7-460c-aee2-af5262da4af4\") " pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:17.823796 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:17.823779 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:17.823883 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:17.823812 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:17.823883 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:17.823826 2572 projected.go:196] Error preparing data for projected volume kube-api-access-fv4qf for pod openshift-network-diagnostics/network-check-target-dgnrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:17.823991 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:17.823888 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf podName:2915194d-a9f7-460c-aee2-af5262da4af4 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:21.823870039 +0000 UTC m=+10.210823928 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fv4qf" (UniqueName: "kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf") pod "network-check-target-dgnrj" (UID: "2915194d-a9f7-460c-aee2-af5262da4af4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:18.120686 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:18.120584 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:18.120849 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:18.120704 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:18.121119 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:18.121100 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:18.121233 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:18.121197 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:18.121298 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:18.121261 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:18.121366 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:18.121342 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:18.221530 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:18.221446 2572 generic.go:356] "Generic (PLEG): container finished" podID="9b663748-9129-4d2c-b900-c5ddde8b4b63" containerID="883447b0e26d5c770faead762bc31a984bb0eb4243330d4631616c13c3050bb8" exitCode=0 Oct 30 12:51:18.221849 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:18.221767 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mmzs" event={"ID":"9b663748-9129-4d2c-b900-c5ddde8b4b63","Type":"ContainerDied","Data":"883447b0e26d5c770faead762bc31a984bb0eb4243330d4631616c13c3050bb8"} Oct 30 12:51:19.230262 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:19.230222 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zchsd" event={"ID":"741107a5-c4db-4ba9-929c-58952377929f","Type":"ContainerStarted","Data":"71b7673da012bda895960fc656e9e3e442f954439b76863ec14d2826302cf4d5"} Oct 30 12:51:20.113952 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:20.113920 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:20.114121 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:20.114047 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:20.114429 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:20.114407 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:20.114607 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:20.114585 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:20.114693 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:20.114650 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:20.114746 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:20.114706 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:20.242571 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:20.242539 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bm8bc" event={"ID":"8bc2e7c1-04ef-439a-90e5-57c4c200c62f","Type":"ContainerStarted","Data":"b24b257852d78c81eb85ece93215bb73dc5e073deaaea2ad48c70037875b81de"} Oct 30 12:51:20.280244 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:20.279972 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zchsd" podStartSLOduration=4.288556919 podStartE2EDuration="8.279957275s" totalImagesPullingTime="3.991400356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:19.286251619 +0000 UTC m=+7.673205515" watchObservedRunningTime="2026-10-30 12:51:20.279957275 +0000 UTC m=+8.666911172" Oct 30 12:51:21.249954 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:21.249914 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" event={"ID":"404bb84d-f044-488e-a221-389f71e61181","Type":"ContainerStarted","Data":"70642ba7bc65f6d1e0cc5eda64d5ac3ec3624bd63da3fe6ca08efc74401850cf"} Oct 30 12:51:21.674019 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:21.673986 2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 12:51:21.760190 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:21.760150 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:21.760347 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:21.760207 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs\") pod \"network-metrics-daemon-b68v5\" (UID: \"050eebb4-7228-4189-b9ec-5bac3d7ed58e\") " pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:21.760347 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:21.760335 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:21.760475 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:21.760391 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs podName:050eebb4-7228-4189-b9ec-5bac3d7ed58e nodeName:}" failed. No retries permitted until 2026-10-30 12:51:29.760378376 +0000 UTC m=+18.147332256 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs") pod "network-metrics-daemon-b68v5" (UID: "050eebb4-7228-4189-b9ec-5bac3d7ed58e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:21.760822 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:21.760800 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:21.760923 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:21.760860 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret podName:0b497ad2-5bd4-4b7b-881d-d94c15fda49c nodeName:}" failed. No retries permitted until 2026-10-30 12:51:29.760844279 +0000 UTC m=+18.147798171 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret") pod "global-pull-secret-syncer-2xg75" (UID: "0b497ad2-5bd4-4b7b-881d-d94c15fda49c") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:21.861644 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:21.861086 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4qf\" (UniqueName: \"kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf\") pod \"network-check-target-dgnrj\" (UID: \"2915194d-a9f7-460c-aee2-af5262da4af4\") " pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:21.861644 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:21.861232 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:21.861644 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:21.861246 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:21.861644 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:21.861255 2572 projected.go:196] Error preparing data for projected volume kube-api-access-fv4qf for pod openshift-network-diagnostics/network-check-target-dgnrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:21.861644 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:21.861296 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf podName:2915194d-a9f7-460c-aee2-af5262da4af4 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:29.861284118 +0000 UTC m=+18.248238006 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fv4qf" (UniqueName: "kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf") pod "network-check-target-dgnrj" (UID: "2915194d-a9f7-460c-aee2-af5262da4af4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:22.045594 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:22.045481 2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:51:21.674014427Z","UUID":"75bee4a7-368c-4231-b598-d1ade91fa600","Handler":null,"Name":"","Endpoint":""} Oct 30 12:51:22.047859 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:22.047833 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 12:51:22.047859 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:22.047867 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 12:51:22.109536 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:22.109505 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:22.109536 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:22.109534 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:22.109738 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:22.109646 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:22.109814 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:22.109765 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:22.109990 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:22.109837 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:22.110257 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:22.110239 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:22.258536 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:22.258500 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" event={"ID":"404bb84d-f044-488e-a221-389f71e61181","Type":"ContainerStarted","Data":"fb618b433e70465d97d8064bf9e45d233aaae08eb194e56d6bafa2856fe632d6"} Oct 30 12:51:22.262689 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:22.262659 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4tbwx" event={"ID":"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9","Type":"ContainerStarted","Data":"bc4fbf6538079b1074545598a1b9b7c68b8bc03c0e69099b08821441b1008fbb"} Oct 30 12:51:22.304740 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:22.304671 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bm8bc" podStartSLOduration=5.107307348 podStartE2EDuration="10.304652592s" totalImagesPullingTime="5.197345244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:20.280230485 +0000 UTC m=+8.667184381" watchObservedRunningTime="2026-10-30 12:51:22.304652592 +0000 UTC m=+10.691606493" Oct 30 12:51:23.267504 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:23.267239 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qldmf" event={"ID":"109981ff-df1e-4279-bc7a-1e2b09f30ad1","Type":"ContainerStarted","Data":"c9e695f55743248c492f565b172a40ca27b7ff56a3661556fb7d251c3866ce23"} Oct 30 12:51:23.272339 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:23.272302 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" event={"ID":"404bb84d-f044-488e-a221-389f71e61181","Type":"ContainerStarted","Data":"0b47d3bedbc579d031994eb35bc18af37a34fa9ee721e54aa71809ea9abc7472"} Oct 30 12:51:23.275670 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:23.275640 2572 generic.go:356] "Generic (PLEG): container finished" podID="9b663748-9129-4d2c-b900-c5ddde8b4b63" containerID="88130d417b5279da3d0120aae1115f03a4011e6f463a42297a6862de0ba7307b" exitCode=0 Oct 30 12:51:23.275809 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:23.275729 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mmzs" event={"ID":"9b663748-9129-4d2c-b900-c5ddde8b4b63","Type":"ContainerDied","Data":"88130d417b5279da3d0120aae1115f03a4011e6f463a42297a6862de0ba7307b"} Oct 30 12:51:23.306916 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:23.306856 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-4tbwx" podStartSLOduration=3.515766969 podStartE2EDuration="11.306841055s" totalImagesPullingTime="7.791074086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:22.305023813 +0000 UTC m=+10.691977711" watchObservedRunningTime="2026-10-30 12:51:23.306841055 +0000 UTC m=+11.693794951" Oct 30 12:51:23.307332 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:23.307297 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-qldmf" podStartSLOduration=3.442371332 podStartE2EDuration="11.307290459s" totalImagesPullingTime="7.864919127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:23.307179494 +0000 UTC m=+11.694133393" watchObservedRunningTime="2026-10-30 12:51:23.307290459 +0000 UTC m=+11.694244361" Oct 30 12:51:23.391949 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:23.391895 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" podStartSLOduration=3.267629441 podStartE2EDuration="11.391880492s" totalImagesPullingTime="8.124251051s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:23.391780224 +0000 UTC m=+11.778734124" watchObservedRunningTime="2026-10-30 12:51:23.391880492 +0000 UTC m=+11.778834388" Oct 30 12:51:23.615104 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:23.615023 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zchsd" Oct 30 12:51:23.615895 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:23.615867 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zchsd" Oct 30 12:51:24.109845 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:24.109810 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:24.109845 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:24.109835 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:24.110084 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:24.109921 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:24.110084 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:24.109986 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:24.110084 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:24.110074 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:24.110232 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:24.110180 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:24.280543 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:24.280504 2572 generic.go:356] "Generic (PLEG): container finished" podID="9b663748-9129-4d2c-b900-c5ddde8b4b63" containerID="3f255ca524212386aeb4ced3023c846626c9153b11f426df68d387624cefbbb2" exitCode=0 Oct 30 12:51:24.280543 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:24.280521 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mmzs" event={"ID":"9b663748-9129-4d2c-b900-c5ddde8b4b63","Type":"ContainerDied","Data":"3f255ca524212386aeb4ced3023c846626c9153b11f426df68d387624cefbbb2"} Oct 30 12:51:24.283867 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:24.283838 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r9nqr" event={"ID":"83545835-dd10-4f8b-8015-25c579bdac3a","Type":"ContainerStarted","Data":"9335fe61bf3e30b7fd56fbe92002170aff69155c2c0fbeb46c9f37eb9f665ea4"} Oct 30 12:51:24.284565 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:24.284547 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zchsd" Oct 30 12:51:24.284953 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:24.284937 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zchsd" Oct 30 12:51:25.288402 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:25.288363 2572 generic.go:356] "Generic (PLEG): container finished" podID="9b663748-9129-4d2c-b900-c5ddde8b4b63" containerID="4028df408a3169f2ee57728e8c745e0dc26415cadade192eea29acf91bf20202" exitCode=0 Oct 30 12:51:25.289099 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:25.288455 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mmzs" event={"ID":"9b663748-9129-4d2c-b900-c5ddde8b4b63","Type":"ContainerDied","Data":"4028df408a3169f2ee57728e8c745e0dc26415cadade192eea29acf91bf20202"} Oct 30 12:51:25.290199 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:25.290168 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" event={"ID":"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd","Type":"ContainerStarted","Data":"ec4aa7da3cb4fa77a00a8a8aa58fcbbf3b9e3689dbbf5bb7a3ff8e7f1fee2be9"} Oct 30 12:51:25.373448 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:25.373333 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" podStartSLOduration=2.9780076380000002 podStartE2EDuration="13.37332005s" totalImagesPullingTime="10.395312412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:25.373162166 +0000 UTC m=+13.760116063" watchObservedRunningTime="2026-10-30 12:51:25.37332005 +0000 UTC m=+13.760273969" Oct 30 12:51:25.410548 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:25.410476 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r9nqr" podStartSLOduration=5.265815256 podStartE2EDuration="13.410463392s" totalImagesPullingTime="8.144648136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:25.409965392 +0000 UTC m=+13.796919330" watchObservedRunningTime="2026-10-30 12:51:25.410463392 +0000 UTC m=+13.797417306" Oct 30 12:51:26.106634 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:26.106597 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:26.106803 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:26.106667 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:26.106803 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:26.106674 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:26.106896 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:26.106797 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:26.106943 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:26.106918 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:26.107020 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:26.106998 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:27.301339 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:27.301113 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mmzs" event={"ID":"9b663748-9129-4d2c-b900-c5ddde8b4b63","Type":"ContainerStarted","Data":"aadc12dcbf14159d5c5971a7384dd5a9231d3fd7ee273d694e75559036c56382"} Oct 30 12:51:28.105720 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:28.105680 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:28.105720 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:28.105708 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:28.105945 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:28.105729 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:28.105945 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:28.105822 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:28.105945 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:28.105916 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:28.106088 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:28.106009 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:28.305970 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:28.305929 2572 generic.go:356] "Generic (PLEG): container finished" podID="9b663748-9129-4d2c-b900-c5ddde8b4b63" containerID="aadc12dcbf14159d5c5971a7384dd5a9231d3fd7ee273d694e75559036c56382" exitCode=0 Oct 30 12:51:28.306341 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:28.305990 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mmzs" event={"ID":"9b663748-9129-4d2c-b900-c5ddde8b4b63","Type":"ContainerDied","Data":"aadc12dcbf14159d5c5971a7384dd5a9231d3fd7ee273d694e75559036c56382"} Oct 30 12:51:29.313921 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:29.313690 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 12:51:29.314296 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:29.314153 2572 generic.go:356] "Generic (PLEG): container finished" podID="fce77b3b-df3e-4d04-945c-68dc11edc61f" containerID="9a37a3ab36d0e68cb97c45df211a97c7ae013550f314c53ef1c76b27075b7437" exitCode=1 Oct 30 12:51:29.314296 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:29.314224 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" event={"ID":"fce77b3b-df3e-4d04-945c-68dc11edc61f","Type":"ContainerStarted","Data":"3b248d25d71980389b0f1a5d651f144cd119be941bd9e2780b38a1291f7c982a"} Oct 30 12:51:29.314372 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:29.314313 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" event={"ID":"fce77b3b-df3e-4d04-945c-68dc11edc61f","Type":"ContainerStarted","Data":"00c7b584ca78d5f85c90c86d689c4f2dfebfabd5f83d9fc8c5b9efa3d5be4d72"} Oct 30 12:51:29.314372 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:29.314349 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" event={"ID":"fce77b3b-df3e-4d04-945c-68dc11edc61f","Type":"ContainerStarted","Data":"2d5779eded08e545f778fbc2ee43911c01bd8574295006ea1c685a966517d196"} Oct 30 12:51:29.314433 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:29.314386 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" event={"ID":"fce77b3b-df3e-4d04-945c-68dc11edc61f","Type":"ContainerStarted","Data":"48e2c7e674ae2363fb1d7164c36edd1479e3aacc1aca4b99e38b8501da980bab"} Oct 30 12:51:29.314476 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:29.314429 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" event={"ID":"fce77b3b-df3e-4d04-945c-68dc11edc61f","Type":"ContainerDied","Data":"9a37a3ab36d0e68cb97c45df211a97c7ae013550f314c53ef1c76b27075b7437"} Oct 30 12:51:29.314476 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:29.314462 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" event={"ID":"fce77b3b-df3e-4d04-945c-68dc11edc61f","Type":"ContainerStarted","Data":"afce8d049c2609d3fab7f3f7be9e6d1b5fbc7fe79569e7b5f5a7b452dbdcb2c8"} Oct 30 12:51:29.317266 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:29.317242 2572 generic.go:356] "Generic (PLEG): container finished" podID="9b663748-9129-4d2c-b900-c5ddde8b4b63" containerID="fd53f8b4fc42c0bf908590bdf9625ccd5b14913b94e5ec9ce864f8e67bd75180" exitCode=0 Oct 30 12:51:29.317379 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:29.317274 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mmzs" event={"ID":"9b663748-9129-4d2c-b900-c5ddde8b4b63","Type":"ContainerDied","Data":"fd53f8b4fc42c0bf908590bdf9625ccd5b14913b94e5ec9ce864f8e67bd75180"} Oct 30 12:51:29.820022 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:29.819985 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:29.820169 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:29.820038 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs\") pod \"network-metrics-daemon-b68v5\" (UID: \"050eebb4-7228-4189-b9ec-5bac3d7ed58e\") " pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:29.820169 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:29.820135 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:29.820169 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:29.820135 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:29.820261 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:29.820192 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs podName:050eebb4-7228-4189-b9ec-5bac3d7ed58e nodeName:}" failed. No retries permitted until 2026-10-30 12:51:45.820175138 +0000 UTC m=+34.207129013 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs") pod "network-metrics-daemon-b68v5" (UID: "050eebb4-7228-4189-b9ec-5bac3d7ed58e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:29.820261 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:29.820207 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret podName:0b497ad2-5bd4-4b7b-881d-d94c15fda49c nodeName:}" failed. No retries permitted until 2026-10-30 12:51:45.82020062 +0000 UTC m=+34.207154495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret") pod "global-pull-secret-syncer-2xg75" (UID: "0b497ad2-5bd4-4b7b-881d-d94c15fda49c") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:29.920563 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:29.920531 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4qf\" (UniqueName: \"kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf\") pod \"network-check-target-dgnrj\" (UID: \"2915194d-a9f7-460c-aee2-af5262da4af4\") " pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:29.920728 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:29.920681 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:29.920728 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:29.920699 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:29.920728 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:29.920709 2572 projected.go:196] Error preparing data for projected volume kube-api-access-fv4qf for pod openshift-network-diagnostics/network-check-target-dgnrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:29.920814 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:29.920759 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf podName:2915194d-a9f7-460c-aee2-af5262da4af4 nodeName:}" failed. No retries permitted until 2026-10-30 12:51:45.920745366 +0000 UTC m=+34.307699244 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fv4qf" (UniqueName: "kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf") pod "network-check-target-dgnrj" (UID: "2915194d-a9f7-460c-aee2-af5262da4af4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:30.106222 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:30.106136 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:30.106366 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:30.106139 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:30.106366 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:30.106243 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:30.106366 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:30.106267 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:30.106366 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:30.106328 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:30.106531 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:30.106412 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:30.322391 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:30.322361 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4mmzs" event={"ID":"9b663748-9129-4d2c-b900-c5ddde8b4b63","Type":"ContainerStarted","Data":"88c15e31ba7f05b23b1a9a002282503f1f18af4366f1b62985adaaf91a0e2180"} Oct 30 12:51:30.367976 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:30.367876 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4mmzs" podStartSLOduration=7.980621891 podStartE2EDuration="18.36786039s" totalImagesPullingTime="10.387238499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:30.36730479 +0000 UTC m=+18.754258688" watchObservedRunningTime="2026-10-30 12:51:30.36786039 +0000 UTC m=+18.754814286" Oct 30 12:51:31.327559 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:31.327533 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 12:51:31.328059 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:31.327872 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" event={"ID":"fce77b3b-df3e-4d04-945c-68dc11edc61f","Type":"ContainerStarted","Data":"81b62fb324c6ab919eb1c5645fd4e1f21b06c7a39372577774722eafa37fa4c9"} Oct 30 12:51:31.992113 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:31.992082 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qldmf_109981ff-df1e-4279-bc7a-1e2b09f30ad1/dns-node-resolver/0.log" Oct 30 12:51:32.107104 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:32.107074 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:32.107258 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:32.107179 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:32.107258 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:32.107187 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:32.107324 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:32.107261 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:32.107324 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:32.107308 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:32.107385 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:32.107361 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:33.088334 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:33.088307 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bm8bc_8bc2e7c1-04ef-439a-90e5-57c4c200c62f/node-ca/0.log" Oct 30 12:51:33.335173 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:33.335149 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 12:51:33.337260 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:33.337224 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" event={"ID":"fce77b3b-df3e-4d04-945c-68dc11edc61f","Type":"ContainerStarted","Data":"a549bde75d95de0998f20b87aef711a2c09c41f15f565d9e3ba27d43fb132fbe"} Oct 30 12:51:33.337820 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:33.337798 2572 scope.go:122] "RemoveContainer" containerID="9a37a3ab36d0e68cb97c45df211a97c7ae013550f314c53ef1c76b27075b7437" Oct 30 12:51:33.338483 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:33.338429 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:33.338570 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:33.338536 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:33.338675 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:33.338661 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:33.354958 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:33.354937 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:33.356352 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:33.356336 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:51:34.106374 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:34.106335 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:34.106374 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:34.106352 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:34.107042 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:34.106461 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:34.107042 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:34.106570 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:34.107042 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:34.106645 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:34.107042 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:34.106725 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:34.342810 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:34.342784 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 12:51:34.343105 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:34.343085 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" event={"ID":"fce77b3b-df3e-4d04-945c-68dc11edc61f","Type":"ContainerStarted","Data":"1795c380e72c457dce49c86dbef6a3efbb07d991a727640f5451f11273275e62"} Oct 30 12:51:36.108582 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:36.108546 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:36.109077 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:36.108546 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:36.109077 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:36.108647 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:36.109077 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:36.108546 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:36.109077 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:36.108741 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:36.109077 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:36.108798 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:38.105813 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:38.105784 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:38.106331 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:38.105794 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:38.106331 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:38.105900 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:38.106331 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:38.105924 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:38.106331 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:38.105991 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:38.106331 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:38.106047 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:40.106503 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:40.106446 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:40.106957 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:40.106446 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:40.106957 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:40.106568 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:40.106957 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:40.106655 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:40.106957 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:40.106455 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:40.106957 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:40.106761 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:42.107040 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:42.107004 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:42.107411 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:42.107140 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:42.107411 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:42.107155 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:42.107411 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:42.107220 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:42.107411 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:42.107227 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:42.107411 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:42.107301 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:44.106053 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:44.106017 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:44.106531 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:44.106017 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:44.106531 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:44.106143 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:44.106531 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:44.106017 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:44.106531 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:44.106183 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:44.106531 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:44.106286 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:45.829363 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:45.829329 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:45.829363 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:45.829378 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs\") pod \"network-metrics-daemon-b68v5\" (UID: \"050eebb4-7228-4189-b9ec-5bac3d7ed58e\") " pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:45.829901 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:45.829507 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:45.829901 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:45.829559 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs podName:050eebb4-7228-4189-b9ec-5bac3d7ed58e nodeName:}" failed. No retries permitted until 2026-10-30 12:52:17.829547414 +0000 UTC m=+66.216501302 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs") pod "network-metrics-daemon-b68v5" (UID: "050eebb4-7228-4189-b9ec-5bac3d7ed58e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 12:51:45.829901 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:45.829507 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:45.829901 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:45.829638 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret podName:0b497ad2-5bd4-4b7b-881d-d94c15fda49c nodeName:}" failed. No retries permitted until 2026-10-30 12:52:17.829624098 +0000 UTC m=+66.216578002 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret") pod "global-pull-secret-syncer-2xg75" (UID: "0b497ad2-5bd4-4b7b-881d-d94c15fda49c") : object "kube-system"/"original-pull-secret" not registered Oct 30 12:51:45.930177 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:45.930144 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4qf\" (UniqueName: \"kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf\") pod \"network-check-target-dgnrj\" (UID: \"2915194d-a9f7-460c-aee2-af5262da4af4\") " pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:45.930326 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:45.930299 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 12:51:45.930326 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:45.930317 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 12:51:45.930393 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:45.930327 2572 projected.go:196] Error preparing data for projected volume kube-api-access-fv4qf for pod openshift-network-diagnostics/network-check-target-dgnrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:45.930393 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:45.930379 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf podName:2915194d-a9f7-460c-aee2-af5262da4af4 nodeName:}" failed. No retries permitted until 2026-10-30 12:52:17.930367306 +0000 UTC m=+66.317321209 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fv4qf" (UniqueName: "kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf") pod "network-check-target-dgnrj" (UID: "2915194d-a9f7-460c-aee2-af5262da4af4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 12:51:46.106248 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:46.106168 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:46.106386 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:46.106168 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:46.106386 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:46.106172 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:46.106386 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:46.106328 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:46.106513 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:46.106412 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:46.106513 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:46.106475 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:48.105978 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:48.105938 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:48.105978 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:48.105960 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:48.106550 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:48.106028 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:48.106550 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:48.106043 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:48.106550 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:48.106107 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:48.106550 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:48.106155 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:50.106464 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:50.106429 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:50.106942 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:50.106429 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:50.106942 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:50.106558 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:50.106942 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:50.106428 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:50.106942 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:50.106603 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:50.106942 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:50.106707 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:52.106532 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:52.106499 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:52.106907 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:52.106572 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:52.106907 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:52.106591 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:52.106907 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:52.106678 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:52.106907 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:52.106705 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:52.106907 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:52.106757 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:54.028836 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:54.028772 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" podStartSLOduration=27.676546295 podStartE2EDuration="42.028758535s" totalImagesPullingTime="14.35221224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:34.45833686 +0000 UTC m=+22.845290758" watchObservedRunningTime="2026-10-30 12:51:54.028758535 +0000 UTC m=+42.415712432" Oct 30 12:51:54.106311 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:54.106279 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:54.106469 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:54.106285 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:54.106469 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:54.106410 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:54.106469 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:54.106284 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:54.106605 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:54.106468 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:54.106605 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:54.106556 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:55.303971 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:55.303670 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2xg75"] Oct 30 12:51:55.304362 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:55.304044 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:55.304362 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:55.304157 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:55.308062 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:55.308023 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dgnrj"] Oct 30 12:51:55.308174 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:55.308148 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:55.308269 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:55.308247 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:55.308781 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:55.308714 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b68v5"] Oct 30 12:51:55.308893 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:55.308868 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:55.308991 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:55.308968 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:57.105737 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:57.105705 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:57.105737 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:57.105726 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:57.106239 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:57.105805 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:57.106239 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:57.105815 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:57.106239 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:57.105875 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:57.106239 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:57.105955 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:58.171386 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:58.171358 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qldmf_109981ff-df1e-4279-bc7a-1e2b09f30ad1/dns-node-resolver/0.log" Oct 30 12:51:59.105752 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:59.105706 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:51:59.105954 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:59.105710 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:51:59.105954 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:59.105838 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgnrj" podUID="2915194d-a9f7-460c-aee2-af5262da4af4" Oct 30 12:51:59.105954 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:59.105726 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:51:59.105954 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:59.105908 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b68v5" podUID="050eebb4-7228-4189-b9ec-5bac3d7ed58e" Oct 30 12:51:59.106172 ip-10-0-138-103 kubenswrapper[2572]: E1030 12:51:59.105972 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2xg75" podUID="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" Oct 30 12:51:59.571590 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:59.571568 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bm8bc_8bc2e7c1-04ef-439a-90e5-57c4c200c62f/node-ca/0.log" Oct 30 12:51:59.935099 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:59.935072 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeReady" Oct 30 12:51:59.935266 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:59.935188 2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 12:51:59.994358 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:59.994325 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4rfmk"] Oct 30 12:51:59.996241 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:59.996214 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4rfmk" Oct 30 12:51:59.997234 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:59.997212 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4hts2"] Oct 30 12:51:59.998975 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:51:59.998960 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:00.016206 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.016175 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 12:52:00.016357 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.016277 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hhkwq\"" Oct 30 12:52:00.019265 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.019245 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 12:52:00.019374 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.019358 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 12:52:00.019409 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.019363 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 12:52:00.019445 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.019429 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 12:52:00.020126 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.020113 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 12:52:00.034348 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.034308 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4rfmk"] Oct 30 12:52:00.035152 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.035126 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4hts2"] Oct 30 12:52:00.035815 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.035793 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mtsz8"] Oct 30 12:52:00.037835 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.037815 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:00.064425 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.064389 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mtsz8"] Oct 30 12:52:00.111087 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.111058 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6vr75\"" Oct 30 12:52:00.111087 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.111059 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 12:52:00.111087 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.111059 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 12:52:00.111302 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.111066 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kq2jq\"" Oct 30 12:52:00.121964 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.121938 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 12:52:00.130225 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.130206 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-tmp-dir\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:00.130285 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.130239 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5bqz\" (UniqueName: \"kubernetes.io/projected/6b8f3882-2505-491f-8139-1905a70a5840-kube-api-access-p5bqz\") pod \"ingress-canary-4rfmk\" (UID: \"6b8f3882-2505-491f-8139-1905a70a5840\") " pod="openshift-ingress-canary/ingress-canary-4rfmk" Oct 30 12:52:00.130285 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.130262 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-config-volume\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:00.130285 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.130282 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09b6fe90-3962-49ef-bc16-486d09469b22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:00.130432 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.130415 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09b6fe90-3962-49ef-bc16-486d09469b22-crio-socket\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:00.130501 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.130472 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrkbx\" (UniqueName: \"kubernetes.io/projected/09b6fe90-3962-49ef-bc16-486d09469b22-kube-api-access-mrkbx\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:00.130542 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.130515 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09b6fe90-3962-49ef-bc16-486d09469b22-data-volume\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:00.130542 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.130530 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b8f3882-2505-491f-8139-1905a70a5840-cert\") pod \"ingress-canary-4rfmk\" (UID: \"6b8f3882-2505-491f-8139-1905a70a5840\") " pod="openshift-ingress-canary/ingress-canary-4rfmk" Oct 30 12:52:00.130601 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.130546 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdpc7\" (UniqueName: \"kubernetes.io/projected/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-kube-api-access-wdpc7\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:00.130601 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.130590 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09b6fe90-3962-49ef-bc16-486d09469b22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:00.130658 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.130615 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-metrics-tls\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:00.231129 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.231032 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrkbx\" (UniqueName: \"kubernetes.io/projected/09b6fe90-3962-49ef-bc16-486d09469b22-kube-api-access-mrkbx\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:00.231129 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.231082 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09b6fe90-3962-49ef-bc16-486d09469b22-data-volume\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:00.231343 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.231201 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b8f3882-2505-491f-8139-1905a70a5840-cert\") pod \"ingress-canary-4rfmk\" (UID: \"6b8f3882-2505-491f-8139-1905a70a5840\") " pod="openshift-ingress-canary/ingress-canary-4rfmk" Oct 30 12:52:00.231343 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.231246 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdpc7\" (UniqueName: \"kubernetes.io/projected/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-kube-api-access-wdpc7\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:00.231343 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.231295 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09b6fe90-3962-49ef-bc16-486d09469b22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:00.231343 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.231318 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-metrics-tls\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:00.231574 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.231344 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-tmp-dir\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:00.231574 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.231376 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5bqz\" (UniqueName: \"kubernetes.io/projected/6b8f3882-2505-491f-8139-1905a70a5840-kube-api-access-p5bqz\") pod \"ingress-canary-4rfmk\" (UID: \"6b8f3882-2505-491f-8139-1905a70a5840\") " pod="openshift-ingress-canary/ingress-canary-4rfmk" Oct 30 12:52:00.231574 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.231429 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09b6fe90-3962-49ef-bc16-486d09469b22-data-volume\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:00.231574 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.231416 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-config-volume\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:00.231574 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.231473 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09b6fe90-3962-49ef-bc16-486d09469b22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:00.231803 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.231590 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09b6fe90-3962-49ef-bc16-486d09469b22-crio-socket\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:00.231803 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.231715 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09b6fe90-3962-49ef-bc16-486d09469b22-crio-socket\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:00.231803 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.231783 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-tmp-dir\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:00.231991 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.231965 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09b6fe90-3962-49ef-bc16-486d09469b22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:00.232071 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.232050 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-config-volume\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:00.235234 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.235210 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09b6fe90-3962-49ef-bc16-486d09469b22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:00.235333 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.235212 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-metrics-tls\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:00.235392 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.235371 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b8f3882-2505-491f-8139-1905a70a5840-cert\") pod \"ingress-canary-4rfmk\" (UID: \"6b8f3882-2505-491f-8139-1905a70a5840\") " pod="openshift-ingress-canary/ingress-canary-4rfmk" Oct 30 12:52:00.254079 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.254042 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrkbx\" (UniqueName: \"kubernetes.io/projected/09b6fe90-3962-49ef-bc16-486d09469b22-kube-api-access-mrkbx\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:00.254079 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.254061 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5bqz\" (UniqueName: \"kubernetes.io/projected/6b8f3882-2505-491f-8139-1905a70a5840-kube-api-access-p5bqz\") pod \"ingress-canary-4rfmk\" (UID: \"6b8f3882-2505-491f-8139-1905a70a5840\") " pod="openshift-ingress-canary/ingress-canary-4rfmk" Oct 30 12:52:00.254278 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.254042 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdpc7\" (UniqueName: \"kubernetes.io/projected/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-kube-api-access-wdpc7\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:00.307843 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.307807 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4rfmk" Oct 30 12:52:00.314690 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.314656 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:00.349043 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.349003 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:00.651533 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.651427 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4rfmk"] Oct 30 12:52:00.655051 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.655024 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4hts2"] Oct 30 12:52:00.657810 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:52:00.657741 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8f3882_2505_491f_8139_1905a70a5840.slice/crio-d751490f72a625e19a73650397f038836285aff2f8e41b56ec64e5b3b80f7ec7 WatchSource:0}: Error finding container d751490f72a625e19a73650397f038836285aff2f8e41b56ec64e5b3b80f7ec7: Status 404 returned error can't find the container with id d751490f72a625e19a73650397f038836285aff2f8e41b56ec64e5b3b80f7ec7 Oct 30 12:52:00.661109 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:52:00.661089 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b6fe90_3962_49ef_bc16_486d09469b22.slice/crio-e1c4b32f926d9b2b17727f862be9d30fb04dd7f0de66dcefadf536d2c3a87092 WatchSource:0}: Error finding container e1c4b32f926d9b2b17727f862be9d30fb04dd7f0de66dcefadf536d2c3a87092: Status 404 returned error can't find the container with id e1c4b32f926d9b2b17727f862be9d30fb04dd7f0de66dcefadf536d2c3a87092 Oct 30 12:52:00.668979 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:00.668956 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mtsz8"] Oct 30 12:52:00.673615 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:52:00.673533 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1069a5ce_5720_4e0d_8bb8_cc874020e3dd.slice/crio-7d83332a9c55d3325c784e5856c29be7531d22e2c24b35dd533ed8cfd1711093 WatchSource:0}: Error finding container 7d83332a9c55d3325c784e5856c29be7531d22e2c24b35dd533ed8cfd1711093: Status 404 returned error can't find the container with id 7d83332a9c55d3325c784e5856c29be7531d22e2c24b35dd533ed8cfd1711093 Oct 30 12:52:01.108083 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:01.106512 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:52:01.108083 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:01.107012 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:52:01.108083 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:01.107038 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:52:01.121126 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:01.121099 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 12:52:01.121321 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:01.121255 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 12:52:01.133015 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:01.132902 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rnjq7\"" Oct 30 12:52:01.133015 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:01.132933 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n54s8\"" Oct 30 12:52:01.133015 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:01.132948 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 12:52:01.133015 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:01.132987 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sb6s2\"" Oct 30 12:52:01.133270 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:01.133053 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 12:52:01.405244 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:01.405209 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4hts2" event={"ID":"09b6fe90-3962-49ef-bc16-486d09469b22","Type":"ContainerStarted","Data":"297769362c26397f623c0f885a9829560fc7252eaec7e703ac57086286cb2902"} Oct 30 12:52:01.405419 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:01.405284 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4hts2" event={"ID":"09b6fe90-3962-49ef-bc16-486d09469b22","Type":"ContainerStarted","Data":"be25c0ec2e7985c388284cc88fd1bbca9c55bcbf25aab105134d4dd0834c1b24"} Oct 30 12:52:01.405419 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:01.405311 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4hts2" event={"ID":"09b6fe90-3962-49ef-bc16-486d09469b22","Type":"ContainerStarted","Data":"e1c4b32f926d9b2b17727f862be9d30fb04dd7f0de66dcefadf536d2c3a87092"} Oct 30 12:52:01.406794 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:01.406769 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mtsz8" event={"ID":"1069a5ce-5720-4e0d-8bb8-cc874020e3dd","Type":"ContainerStarted","Data":"7d83332a9c55d3325c784e5856c29be7531d22e2c24b35dd533ed8cfd1711093"} Oct 30 12:52:01.408206 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:01.408172 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4rfmk" event={"ID":"6b8f3882-2505-491f-8139-1905a70a5840","Type":"ContainerStarted","Data":"d751490f72a625e19a73650397f038836285aff2f8e41b56ec64e5b3b80f7ec7"} Oct 30 12:52:02.327380 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.327346 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg"] Oct 30 12:52:02.334444 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.333359 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:02.353736 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.353554 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg"] Oct 30 12:52:02.353736 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.353556 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 12:52:02.353736 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.353696 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 12:52:02.355479 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.355457 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 12:52:02.356967 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.355566 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 12:52:02.356967 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.356610 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-22g27\"" Oct 30 12:52:02.356967 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.356713 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 12:52:02.380024 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.379987 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf"] Oct 30 12:52:02.382474 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.382444 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.400692 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.400661 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vdft6"] Oct 30 12:52:02.403048 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.403022 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.411733 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.410866 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf"] Oct 30 12:52:02.449563 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.449529 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31e0a09f-ff45-482d-afec-c57bd2d41cb4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:02.449730 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.449576 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31e0a09f-ff45-482d-afec-c57bd2d41cb4-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:02.449730 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.449634 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31e0a09f-ff45-482d-afec-c57bd2d41cb4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:02.449730 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.449707 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xdcg\" (UniqueName: \"kubernetes.io/projected/31e0a09f-ff45-482d-afec-c57bd2d41cb4-kube-api-access-9xdcg\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:02.458186 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.457748 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hxn87\"" Oct 30 12:52:02.458186 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.457846 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 12:52:02.458186 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.457941 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-sn4nb\"" Oct 30 12:52:02.458186 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.458014 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 12:52:02.458186 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.458053 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 12:52:02.458186 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.458113 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 12:52:02.458186 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.458151 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 12:52:02.458608 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.458354 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 12:52:02.551542 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550273 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-root\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.551542 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550314 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8hp2\" (UniqueName: \"kubernetes.io/projected/60a88a61-3c25-48c8-84c5-b35bab4a058a-kube-api-access-p8hp2\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.551542 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550345 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.551542 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550380 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31e0a09f-ff45-482d-afec-c57bd2d41cb4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:02.551542 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550408 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.551542 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550437 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.551542 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550523 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xdcg\" (UniqueName: \"kubernetes.io/projected/31e0a09f-ff45-482d-afec-c57bd2d41cb4-kube-api-access-9xdcg\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:02.551542 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550553 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.551542 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550582 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.551542 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550601 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-tls\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.551542 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550621 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7llpr\" (UniqueName: \"kubernetes.io/projected/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-api-access-7llpr\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.551542 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550638 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-sys\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.551542 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550656 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60a88a61-3c25-48c8-84c5-b35bab4a058a-metrics-client-ca\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.551542 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550673 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.551542 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550692 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-wtmp\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.552333 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550712 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-textfile\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.552333 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550735 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31e0a09f-ff45-482d-afec-c57bd2d41cb4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:02.552333 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550760 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31e0a09f-ff45-482d-afec-c57bd2d41cb4-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:02.552333 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.550780 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.552333 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.552256 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31e0a09f-ff45-482d-afec-c57bd2d41cb4-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:02.554181 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.554132 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31e0a09f-ff45-482d-afec-c57bd2d41cb4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:02.556041 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.555987 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31e0a09f-ff45-482d-afec-c57bd2d41cb4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:02.606110 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.606022 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xdcg\" (UniqueName: \"kubernetes.io/projected/31e0a09f-ff45-482d-afec-c57bd2d41cb4-kube-api-access-9xdcg\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:02.651999 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.651879 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60a88a61-3c25-48c8-84c5-b35bab4a058a-metrics-client-ca\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.651999 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.651924 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.651999 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.651953 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-wtmp\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.651999 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.651983 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-textfile\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.652350 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.652020 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.652350 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.652058 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-root\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.652350 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.652082 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8hp2\" (UniqueName: \"kubernetes.io/projected/60a88a61-3c25-48c8-84c5-b35bab4a058a-kube-api-access-p8hp2\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.652350 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.652108 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.652350 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.652146 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.652350 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.652192 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.652350 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.652260 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.652350 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.652294 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.652350 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.652326 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-tls\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.652846 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.652390 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7llpr\" (UniqueName: \"kubernetes.io/projected/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-api-access-7llpr\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.652846 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.652411 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-sys\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.652846 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.652456 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-textfile\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.652846 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.652544 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-sys\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.653166 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.653144 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60a88a61-3c25-48c8-84c5-b35bab4a058a-metrics-client-ca\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.653237 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.653194 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-root\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.653531 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.653460 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.653633 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.653615 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-wtmp\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.654022 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.653997 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.654458 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.654435 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.654550 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.654519 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.655437 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.655406 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.656765 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.656744 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-tls\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.658510 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.658474 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.658666 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.658604 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:02.661013 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.660955 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.699840 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.699804 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8hp2\" (UniqueName: \"kubernetes.io/projected/60a88a61-3c25-48c8-84c5-b35bab4a058a-kube-api-access-p8hp2\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.701565 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.701518 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7llpr\" (UniqueName: \"kubernetes.io/projected/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-api-access-7llpr\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.706415 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.706255 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:02.719346 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.719319 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:02.737512 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:52:02.735142 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a88a61_3c25_48c8_84c5_b35bab4a058a.slice/crio-cece46519ef9d1fce8d5bb86eafa8086a24c14bf3a983d78738900b8eace0b0f WatchSource:0}: Error finding container cece46519ef9d1fce8d5bb86eafa8086a24c14bf3a983d78738900b8eace0b0f: Status 404 returned error can't find the container with id cece46519ef9d1fce8d5bb86eafa8086a24c14bf3a983d78738900b8eace0b0f Oct 30 12:52:02.989228 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:02.989196 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg"] Oct 30 12:52:02.993536 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:52:02.993505 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e0a09f_ff45_482d_afec_c57bd2d41cb4.slice/crio-cd9622a5d40c0137c96cff0882970b2c6cb0a143371074a75280c334383125fd WatchSource:0}: Error finding container cd9622a5d40c0137c96cff0882970b2c6cb0a143371074a75280c334383125fd: Status 404 returned error can't find the container with id cd9622a5d40c0137c96cff0882970b2c6cb0a143371074a75280c334383125fd Oct 30 12:52:03.038902 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.038874 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf"] Oct 30 12:52:03.039055 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:52:03.039035 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a0e1e1_6da5_4e81_8e1f_ac1643fd0ca1.slice/crio-e8176bf5fb9c579053d7eacdc09d0fcc59da745c7adc6c811ae87881f314db32 WatchSource:0}: Error finding container e8176bf5fb9c579053d7eacdc09d0fcc59da745c7adc6c811ae87881f314db32: Status 404 returned error can't find the container with id e8176bf5fb9c579053d7eacdc09d0fcc59da745c7adc6c811ae87881f314db32 Oct 30 12:52:03.422584 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.422540 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4hts2" event={"ID":"09b6fe90-3962-49ef-bc16-486d09469b22","Type":"ContainerStarted","Data":"0bb4c16e9f17525d4a0625832db4e172f02091c1d672c8745f54dcb563bfd133"} Oct 30 12:52:03.426069 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.426029 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vdft6" event={"ID":"60a88a61-3c25-48c8-84c5-b35bab4a058a","Type":"ContainerStarted","Data":"cece46519ef9d1fce8d5bb86eafa8086a24c14bf3a983d78738900b8eace0b0f"} Oct 30 12:52:03.427621 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.427593 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" event={"ID":"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1","Type":"ContainerStarted","Data":"e8176bf5fb9c579053d7eacdc09d0fcc59da745c7adc6c811ae87881f314db32"} Oct 30 12:52:03.430567 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.430533 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" event={"ID":"31e0a09f-ff45-482d-afec-c57bd2d41cb4","Type":"ContainerStarted","Data":"91cff5c1a677ed9823681f597e2d959db4c8f0f2e7462ff359361ed34ddf5d5d"} Oct 30 12:52:03.430677 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.430594 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" event={"ID":"31e0a09f-ff45-482d-afec-c57bd2d41cb4","Type":"ContainerStarted","Data":"1981ef628be32c5a534c8acf70b75f5fae12acc6bbe2516fb1d402a57e7284af"} Oct 30 12:52:03.430677 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.430624 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" event={"ID":"31e0a09f-ff45-482d-afec-c57bd2d41cb4","Type":"ContainerStarted","Data":"cd9622a5d40c0137c96cff0882970b2c6cb0a143371074a75280c334383125fd"} Oct 30 12:52:03.434568 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.434092 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mtsz8" event={"ID":"1069a5ce-5720-4e0d-8bb8-cc874020e3dd","Type":"ContainerStarted","Data":"ca52d5d691e60256ec7f326c801555d3197a1ee175c2fa89c9c68925d83424c0"} Oct 30 12:52:03.434568 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.434148 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mtsz8" event={"ID":"1069a5ce-5720-4e0d-8bb8-cc874020e3dd","Type":"ContainerStarted","Data":"ec0630cbab363fe8cf192b732dbeada583d8703dcd0bc18859ebed7cfc2dccdc"} Oct 30 12:52:03.434568 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.434483 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:03.469421 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.469342 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4hts2" podStartSLOduration=2.373771986 podStartE2EDuration="4.469323986s" totalImagesPullingTime="2.095552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:51:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:03.468338951 +0000 UTC m=+51.855292917" watchObservedRunningTime="2026-10-30 12:52:03.469323986 +0000 UTC m=+51.856277883" Oct 30 12:52:03.508258 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.508199 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-mtsz8" podStartSLOduration=2.755889118 podStartE2EDuration="4.508180072s" totalImagesPullingTime="1.752290954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:03.50784494 +0000 UTC m=+51.894798856" watchObservedRunningTime="2026-10-30 12:52:03.508180072 +0000 UTC m=+51.895133971" Oct 30 12:52:03.759552 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.759516 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:03.762518 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.762476 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.777058 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.777034 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 12:52:03.779925 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.779901 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 12:52:03.780040 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.779984 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 12:52:03.780094 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.780082 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 12:52:03.780138 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.780132 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 12:52:03.780167 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.780157 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 12:52:03.780222 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.780213 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 12:52:03.780284 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.780272 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 12:52:03.791994 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.791942 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:03.792305 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.792287 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 12:52:03.792367 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.792349 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6lqrk\"" Oct 30 12:52:03.862861 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.862825 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.863022 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.862875 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.863022 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.862906 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.863022 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.862929 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.863022 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.862944 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.863022 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.862963 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.863022 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.862979 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-web-config\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.863022 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.863004 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-volume\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.863022 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.863022 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-out\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.863274 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.863037 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2wdq\" (UniqueName: \"kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-kube-api-access-h2wdq\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.863274 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.863059 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.863274 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.863076 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.863274 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.863098 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.963806 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.963772 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.963949 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.963865 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.963949 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.963899 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.963949 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.963926 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.964085 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.963966 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.964085 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.963991 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.964085 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.964036 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.964085 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.964059 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-web-config\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.964221 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.964096 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-volume\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.964221 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.964123 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-out\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.964221 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.964154 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2wdq\" (UniqueName: \"kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-kube-api-access-h2wdq\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.964221 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.964194 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.964349 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.964217 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.964999 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.964975 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.966388 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.966091 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.967330 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.967286 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.969834 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.969807 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-out\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.969921 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.969856 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.969921 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.969899 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.970341 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.970321 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-web-config\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.970418 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.970396 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.970722 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.970698 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.971016 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.970994 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.971822 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.971800 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.971900 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.971880 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-volume\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:03.990398 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:03.990368 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2wdq\" (UniqueName: \"kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-kube-api-access-h2wdq\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:04.085366 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.085330 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:04.439618 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.439574 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4rfmk" event={"ID":"6b8f3882-2505-491f-8139-1905a70a5840","Type":"ContainerStarted","Data":"4afd0b7eae694b3eac4bf96d7b97c32029df175de5694e0bf09ce55543a5f1f2"} Oct 30 12:52:04.441924 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.441892 2572 generic.go:356] "Generic (PLEG): container finished" podID="60a88a61-3c25-48c8-84c5-b35bab4a058a" containerID="9ae6070703f12858dc766936e59bec535b53fa4de51d52e074ce3edf46b7bd82" exitCode=0 Oct 30 12:52:04.442077 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.442005 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vdft6" event={"ID":"60a88a61-3c25-48c8-84c5-b35bab4a058a","Type":"ContainerDied","Data":"9ae6070703f12858dc766936e59bec535b53fa4de51d52e074ce3edf46b7bd82"} Oct 30 12:52:04.444901 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.444869 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" event={"ID":"31e0a09f-ff45-482d-afec-c57bd2d41cb4","Type":"ContainerStarted","Data":"b6c7c008d8fce84c939ac15d6c23963abe44d2199c779dcdb5884cc86d0d9af1"} Oct 30 12:52:04.473011 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.472983 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:04.476976 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:52:04.476945 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f20c3e2_7ab8_4b88_9e8e_5e59023f1269.slice/crio-9b05c2ce173dda96624c02c816fa0f7b5b5baeca0930eb0255007fd6da3fe6dd WatchSource:0}: Error finding container 9b05c2ce173dda96624c02c816fa0f7b5b5baeca0930eb0255007fd6da3fe6dd: Status 404 returned error can't find the container with id 9b05c2ce173dda96624c02c816fa0f7b5b5baeca0930eb0255007fd6da3fe6dd Oct 30 12:52:04.498709 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.498655 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4rfmk" podStartSLOduration=2.605382092 podStartE2EDuration="5.498635922s" totalImagesPullingTime="2.89325383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:04.497210356 +0000 UTC m=+52.884164247" watchObservedRunningTime="2026-10-30 12:52:04.498635922 +0000 UTC m=+52.885589821" Oct 30 12:52:04.540442 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.540400 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" podStartSLOduration=1.522973204 podStartE2EDuration="2.540384701s" totalImagesPullingTime="1.017411497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:04.539673603 +0000 UTC m=+52.926627500" watchObservedRunningTime="2026-10-30 12:52:04.540384701 +0000 UTC m=+52.927338598" Oct 30 12:52:04.564970 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.564934 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-79ff6956fb-ljgd2"] Oct 30 12:52:04.567786 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.567765 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.588839 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.588817 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-l6ww5\"" Oct 30 12:52:04.588970 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.588849 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-am8n6hsri9i07\"" Oct 30 12:52:04.588970 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.588849 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 12:52:04.588970 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.588872 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 12:52:04.588970 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.588856 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 12:52:04.588970 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.588907 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 12:52:04.588970 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.588941 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 12:52:04.593297 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.593127 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79ff6956fb-ljgd2"] Oct 30 12:52:04.671678 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.671641 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-tls\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.671815 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.671691 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.671815 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.671790 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8915de45-eb8f-451b-8011-2cbf588de9b5-tmp\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.671877 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.671826 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.671877 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.671859 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.671934 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.671887 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2r9g\" (UniqueName: \"kubernetes.io/projected/8915de45-eb8f-451b-8011-2cbf588de9b5-kube-api-access-n2r9g\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.671968 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.671937 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.671998 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.671975 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8915de45-eb8f-451b-8011-2cbf588de9b5-metrics-client-ca\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.672124 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.672042 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-grpc-tls\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.773707 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.773654 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8915de45-eb8f-451b-8011-2cbf588de9b5-metrics-client-ca\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.773904 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.773779 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-grpc-tls\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.773904 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.773825 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-tls\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.773904 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.773846 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.773904 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.773883 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8915de45-eb8f-451b-8011-2cbf588de9b5-tmp\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.774114 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.773908 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.774114 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.773947 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.774114 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.773978 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2r9g\" (UniqueName: \"kubernetes.io/projected/8915de45-eb8f-451b-8011-2cbf588de9b5-kube-api-access-n2r9g\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.774114 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.774032 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.774352 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.774327 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8915de45-eb8f-451b-8011-2cbf588de9b5-tmp\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.774631 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.774606 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8915de45-eb8f-451b-8011-2cbf588de9b5-metrics-client-ca\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.777156 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.777127 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.777275 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.777184 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-grpc-tls\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.777571 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.777550 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.777647 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.777621 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.777778 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.777760 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.777809 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.777787 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-tls\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.797999 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.797966 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2r9g\" (UniqueName: \"kubernetes.io/projected/8915de45-eb8f-451b-8011-2cbf588de9b5-kube-api-access-n2r9g\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:04.895603 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:04.895567 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:05.212711 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:05.212620 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79ff6956fb-ljgd2"] Oct 30 12:52:05.217142 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:52:05.217111 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8915de45_eb8f_451b_8011_2cbf588de9b5.slice/crio-2159f410d7248ba49f61a9f95c4b8a0a641d6137854f345887de4ed27c15e497 WatchSource:0}: Error finding container 2159f410d7248ba49f61a9f95c4b8a0a641d6137854f345887de4ed27c15e497: Status 404 returned error can't find the container with id 2159f410d7248ba49f61a9f95c4b8a0a641d6137854f345887de4ed27c15e497 Oct 30 12:52:05.357150 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:05.357115 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:05.448930 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:05.448890 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269","Type":"ContainerStarted","Data":"9b05c2ce173dda96624c02c816fa0f7b5b5baeca0930eb0255007fd6da3fe6dd"} Oct 30 12:52:05.451645 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:05.451609 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" event={"ID":"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1","Type":"ContainerStarted","Data":"b80a60fb1b569ff53faaf2c71c360877bc2a4c29d40b9d4d078250d639cd81d7"} Oct 30 12:52:05.451777 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:05.451675 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" event={"ID":"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1","Type":"ContainerStarted","Data":"3adfe1d6ca2968ab8c4b6261eafdcf5428e52926c88b76dac6a8ae124416e3fa"} Oct 30 12:52:05.451777 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:05.451710 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" event={"ID":"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1","Type":"ContainerStarted","Data":"0ce54910df49eebaa3512325760425c878f6258f9fd08437ce22f0ecfe6da4b5"} Oct 30 12:52:05.454153 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:05.454127 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vdft6" event={"ID":"60a88a61-3c25-48c8-84c5-b35bab4a058a","Type":"ContainerStarted","Data":"66afb6a76a6ffe9966cef6030809befd2d1c248de08682fb4c76498ee4e2284d"} Oct 30 12:52:05.454294 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:05.454176 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vdft6" event={"ID":"60a88a61-3c25-48c8-84c5-b35bab4a058a","Type":"ContainerStarted","Data":"5d8064c0600e9d554eab1ec8f059f42ac43a86f2a3daf4cac8a6655e0697d5e5"} Oct 30 12:52:05.455350 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:05.455326 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" event={"ID":"8915de45-eb8f-451b-8011-2cbf588de9b5","Type":"ContainerStarted","Data":"2159f410d7248ba49f61a9f95c4b8a0a641d6137854f345887de4ed27c15e497"} Oct 30 12:52:05.550859 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:05.550770 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" podStartSLOduration=2.079358629 podStartE2EDuration="3.550754327s" totalImagesPullingTime="1.471395698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:05.502995404 +0000 UTC m=+53.889949325" watchObservedRunningTime="2026-10-30 12:52:05.550754327 +0000 UTC m=+53.937708224" Oct 30 12:52:05.551369 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:05.551342 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vdft6" podStartSLOduration=2.269866625 podStartE2EDuration="3.551333867s" totalImagesPullingTime="1.200915181s" totalInitContainerRuntime="80.552061ms" isStatefulPod=true podCreationTimestamp="2026-10-30 12:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:05.550215457 +0000 UTC m=+53.937169356" watchObservedRunningTime="2026-10-30 12:52:05.551333867 +0000 UTC m=+53.938287764" Oct 30 12:52:06.460255 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:06.460223 2572 generic.go:356] "Generic (PLEG): container finished" podID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerID="9c17b806405334125a9c015df1ccb6aaf991fcea9c980a47541152a5fd6c8908" exitCode=0 Oct 30 12:52:06.460687 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:06.460310 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269","Type":"ContainerDied","Data":"9c17b806405334125a9c015df1ccb6aaf991fcea9c980a47541152a5fd6c8908"} Oct 30 12:52:07.164928 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.164896 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-56b76dcb6d-fm86z"] Oct 30 12:52:07.167892 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.167866 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.170561 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.170537 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl"] Oct 30 12:52:07.172735 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.172717 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" Oct 30 12:52:07.193866 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.193157 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 12:52:07.193866 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.193207 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 12:52:07.203195 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.203045 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl"] Oct 30 12:52:07.203605 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.203513 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 12:52:07.203810 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.203638 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 12:52:07.204239 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.204157 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-767qk\"" Oct 30 12:52:07.204239 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.204197 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-wqldm\"" Oct 30 12:52:07.207015 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.206993 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-60o5qmh2r8rg1\"" Oct 30 12:52:07.207211 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.207194 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 12:52:07.208895 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.208856 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-56b76dcb6d-fm86z"] Oct 30 12:52:07.299016 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.298979 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdp4l\" (UniqueName: \"kubernetes.io/projected/bb39553b-ccf2-4cb0-af1c-339373ec1b13-kube-api-access-bdp4l\") pod \"monitoring-plugin-77f4f76594-g2zfl\" (UID: \"bb39553b-ccf2-4cb0-af1c-339373ec1b13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" Oct 30 12:52:07.299151 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.299036 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5b885e3c-593b-45d1-b4d4-4604ca511662-metrics-server-audit-profiles\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.299151 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.299115 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-client-ca-bundle\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.299217 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.299148 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flgvz\" (UniqueName: \"kubernetes.io/projected/5b885e3c-593b-45d1-b4d4-4604ca511662-kube-api-access-flgvz\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.299217 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.299179 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb39553b-ccf2-4cb0-af1c-339373ec1b13-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-g2zfl\" (UID: \"bb39553b-ccf2-4cb0-af1c-339373ec1b13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" Oct 30 12:52:07.299272 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.299228 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5b885e3c-593b-45d1-b4d4-4604ca511662-audit-log\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.299272 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.299259 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-secret-metrics-server-client-certs\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.299333 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.299295 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-secret-metrics-server-tls\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.299364 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.299333 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b885e3c-593b-45d1-b4d4-4604ca511662-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.400269 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.400187 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-secret-metrics-server-tls\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.400269 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.400232 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b885e3c-593b-45d1-b4d4-4604ca511662-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.400269 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.400257 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdp4l\" (UniqueName: \"kubernetes.io/projected/bb39553b-ccf2-4cb0-af1c-339373ec1b13-kube-api-access-bdp4l\") pod \"monitoring-plugin-77f4f76594-g2zfl\" (UID: \"bb39553b-ccf2-4cb0-af1c-339373ec1b13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" Oct 30 12:52:07.400692 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.400663 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5b885e3c-593b-45d1-b4d4-4604ca511662-metrics-server-audit-profiles\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.400788 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.400755 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-client-ca-bundle\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.400835 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.400796 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flgvz\" (UniqueName: \"kubernetes.io/projected/5b885e3c-593b-45d1-b4d4-4604ca511662-kube-api-access-flgvz\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.400835 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.400828 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb39553b-ccf2-4cb0-af1c-339373ec1b13-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-g2zfl\" (UID: \"bb39553b-ccf2-4cb0-af1c-339373ec1b13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" Oct 30 12:52:07.400932 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.400878 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5b885e3c-593b-45d1-b4d4-4604ca511662-audit-log\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.400932 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.400905 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-secret-metrics-server-client-certs\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.401060 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.401037 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b885e3c-593b-45d1-b4d4-4604ca511662-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.401393 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.401325 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5b885e3c-593b-45d1-b4d4-4604ca511662-audit-log\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.401707 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.401685 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5b885e3c-593b-45d1-b4d4-4604ca511662-metrics-server-audit-profiles\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.403165 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.403145 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-secret-metrics-server-tls\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.403605 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.403587 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb39553b-ccf2-4cb0-af1c-339373ec1b13-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-g2zfl\" (UID: \"bb39553b-ccf2-4cb0-af1c-339373ec1b13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" Oct 30 12:52:07.403710 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.403693 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-secret-metrics-server-client-certs\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.405954 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.405929 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-client-ca-bundle\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.423238 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.423213 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flgvz\" (UniqueName: \"kubernetes.io/projected/5b885e3c-593b-45d1-b4d4-4604ca511662-kube-api-access-flgvz\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.423414 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.423388 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdp4l\" (UniqueName: \"kubernetes.io/projected/bb39553b-ccf2-4cb0-af1c-339373ec1b13-kube-api-access-bdp4l\") pod \"monitoring-plugin-77f4f76594-g2zfl\" (UID: \"bb39553b-ccf2-4cb0-af1c-339373ec1b13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" Oct 30 12:52:07.466395 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.466360 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" event={"ID":"8915de45-eb8f-451b-8011-2cbf588de9b5","Type":"ContainerStarted","Data":"45e5a5222b07d73bf4c81ea91a763e71750cf7e10dff73b8bed6531c7813d581"} Oct 30 12:52:07.466801 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.466425 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" event={"ID":"8915de45-eb8f-451b-8011-2cbf588de9b5","Type":"ContainerStarted","Data":"d5d364ce71fd75c8dd57a701f2b1516016c53c907454c4d64692d3f3b947ca6f"} Oct 30 12:52:07.466801 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.466451 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" event={"ID":"8915de45-eb8f-451b-8011-2cbf588de9b5","Type":"ContainerStarted","Data":"76c89fd17d8d6b91ed9c7dc4be0d3408599cee0dcfc5f990a600fa68cb6b3843"} Oct 30 12:52:07.510679 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.510644 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:07.512063 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.512043 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" Oct 30 12:52:07.836201 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.836115 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl"] Oct 30 12:52:07.839687 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:07.838919 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-56b76dcb6d-fm86z"] Oct 30 12:52:07.843866 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:52:07.843834 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb39553b_ccf2_4cb0_af1c_339373ec1b13.slice/crio-f142aab910f0986998f5a4fdb8b0bc68f393b6aaee1b89d1cbe79f308e2195af WatchSource:0}: Error finding container f142aab910f0986998f5a4fdb8b0bc68f393b6aaee1b89d1cbe79f308e2195af: Status 404 returned error can't find the container with id f142aab910f0986998f5a4fdb8b0bc68f393b6aaee1b89d1cbe79f308e2195af Oct 30 12:52:07.848311 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:52:07.848284 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b885e3c_593b_45d1_b4d4_4604ca511662.slice/crio-3b50ebe230b9d7b6f04e8d6ea27abcbb07763957be846c73dbf54938777b177c WatchSource:0}: Error finding container 3b50ebe230b9d7b6f04e8d6ea27abcbb07763957be846c73dbf54938777b177c: Status 404 returned error can't find the container with id 3b50ebe230b9d7b6f04e8d6ea27abcbb07763957be846c73dbf54938777b177c Oct 30 12:52:08.475339 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:08.475308 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" event={"ID":"8915de45-eb8f-451b-8011-2cbf588de9b5","Type":"ContainerStarted","Data":"d7f1a59569f2d8010721d8466fbcb66f728b07018e59711552402a73092da8b5"} Oct 30 12:52:08.475759 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:08.475373 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" event={"ID":"8915de45-eb8f-451b-8011-2cbf588de9b5","Type":"ContainerStarted","Data":"69ed2813ff861826e6b959fb91259587e16312d63988c91878df4af435c01921"} Oct 30 12:52:08.475759 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:08.475398 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" event={"ID":"8915de45-eb8f-451b-8011-2cbf588de9b5","Type":"ContainerStarted","Data":"69d008807498044fcf90a2b3b341b64bc0fe1260e92e188018a9c749161091da"} Oct 30 12:52:08.476298 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:08.476278 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:08.481251 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:08.481225 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269","Type":"ContainerStarted","Data":"27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde"} Oct 30 12:52:08.481600 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:08.481571 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269","Type":"ContainerStarted","Data":"897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56"} Oct 30 12:52:08.481661 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:08.481636 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269","Type":"ContainerStarted","Data":"3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d"} Oct 30 12:52:08.481694 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:08.481677 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269","Type":"ContainerStarted","Data":"9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc"} Oct 30 12:52:08.481726 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:08.481716 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269","Type":"ContainerStarted","Data":"f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70"} Oct 30 12:52:08.484765 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:08.483680 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" event={"ID":"5b885e3c-593b-45d1-b4d4-4604ca511662","Type":"ContainerStarted","Data":"3b50ebe230b9d7b6f04e8d6ea27abcbb07763957be846c73dbf54938777b177c"} Oct 30 12:52:08.486140 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:08.486083 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" event={"ID":"bb39553b-ccf2-4cb0-af1c-339373ec1b13","Type":"ContainerStarted","Data":"f142aab910f0986998f5a4fdb8b0bc68f393b6aaee1b89d1cbe79f308e2195af"} Oct 30 12:52:08.530562 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:08.530468 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" podStartSLOduration=1.859442709 podStartE2EDuration="4.530447043s" totalImagesPullingTime="2.671004334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:08.52877227 +0000 UTC m=+56.915726192" watchObservedRunningTime="2026-10-30 12:52:08.530447043 +0000 UTC m=+56.917400941" Oct 30 12:52:09.495015 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.494929 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269","Type":"ContainerStarted","Data":"adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08"} Oct 30 12:52:09.496550 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.496524 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" event={"ID":"5b885e3c-593b-45d1-b4d4-4604ca511662","Type":"ContainerStarted","Data":"1f0f7f71fac9b015720785f6308adfa24abe3a0c9a2547ffe9be3a18c6875ebb"} Oct 30 12:52:09.498016 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.497992 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" event={"ID":"bb39553b-ccf2-4cb0-af1c-339373ec1b13","Type":"ContainerStarted","Data":"d8f47b56275f013e72ba4868b106f53ca827a555b435bd0e38147c16586b18e5"} Oct 30 12:52:09.499736 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.498313 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" Oct 30 12:52:09.503301 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.503277 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:09.507391 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.507363 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" Oct 30 12:52:09.507516 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.507465 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.519055 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.519026 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.519175 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.519091 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.519244 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.519184 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.519244 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.519211 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9dqr\" (UniqueName: \"kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-kube-api-access-k9dqr\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.519345 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.519291 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.519520 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.519481 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.519660 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.519637 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.519765 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.519743 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.519815 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.519785 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.519853 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.519817 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.519853 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.519837 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.519962 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.519943 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.520109 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.520082 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-config\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.520177 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.520117 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.520177 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.520162 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.520303 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.520285 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.520405 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.520382 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.520477 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.520462 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.520599 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.520581 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.520673 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.520653 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.537399 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.537362 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 12:52:09.537529 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.537439 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 12:52:09.538109 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.538096 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 12:52:09.563139 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.563105 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 12:52:09.563310 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.563248 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 12:52:09.563310 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.563249 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 12:52:09.563310 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.563308 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 12:52:09.563531 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.563339 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 12:52:09.563531 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.563395 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-27xf9\"" Oct 30 12:52:09.563531 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.563403 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 12:52:09.563531 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.563337 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 12:52:09.563531 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.563422 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 12:52:09.563531 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.563433 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 12:52:09.563531 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.563470 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 12:52:09.563531 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.563510 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ee888a1mr1r2m\"" Oct 30 12:52:09.563811 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.563516 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 12:52:09.564734 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.564712 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:09.589043 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.588981 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.924885268 podStartE2EDuration="6.588961886s" totalImagesPullingTime="2.654193117s" totalInitContainerRuntime="9.883501ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:09.5876249 +0000 UTC m=+57.974578816" watchObservedRunningTime="2026-10-30 12:52:09.588961886 +0000 UTC m=+57.975915796" Oct 30 12:52:09.621043 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621006 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621227 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621052 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621284 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621227 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621284 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621273 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621390 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621312 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621390 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621346 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621390 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621370 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9dqr\" (UniqueName: \"kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-kube-api-access-k9dqr\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621544 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621407 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621544 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621472 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621544 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621530 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621700 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621563 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621700 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621593 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621700 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621619 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621700 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621644 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621700 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621682 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621953 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621723 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-config\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621953 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621754 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621953 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621783 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621953 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621812 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.621953 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.621838 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.622352 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.622326 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.623187 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.623159 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.623353 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.623330 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.623518 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.623478 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.626309 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.626280 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.626521 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.626481 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-config\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.626944 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.626850 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.627520 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.627168 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.627520 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.627264 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.627520 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.627267 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.627520 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.627403 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.627520 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.627424 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.627814 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.627566 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.627814 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.627789 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.628129 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.628081 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.628283 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.628247 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.629182 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.629155 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.629961 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.629938 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.630595 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.630577 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.645723 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.645665 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" podStartSLOduration=1.146682687 podStartE2EDuration="2.645644457s" totalImagesPullingTime="1.49896177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:09.643814595 +0000 UTC m=+58.030768493" watchObservedRunningTime="2026-10-30 12:52:09.645644457 +0000 UTC m=+58.032598358" Oct 30 12:52:09.653058 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.653033 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9dqr\" (UniqueName: \"kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-kube-api-access-k9dqr\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:09.719906 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.719848 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" podStartSLOduration=1.3341058270000001 podStartE2EDuration="2.719828457s" totalImagesPullingTime="1.38572263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:09.71736805 +0000 UTC m=+58.104321947" watchObservedRunningTime="2026-10-30 12:52:09.719828457 +0000 UTC m=+58.106782368" Oct 30 12:52:09.835532 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:09.835430 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:10.193821 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:10.193773 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:10.198337 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:52:10.198303 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd51364d_3838_4a3d_b470_ccf6483121e7.slice/crio-cbad41beb16322a7113878ce13e65fa3aec8536102496e9412a71a69f9be6a4f WatchSource:0}: Error finding container cbad41beb16322a7113878ce13e65fa3aec8536102496e9412a71a69f9be6a4f: Status 404 returned error can't find the container with id cbad41beb16322a7113878ce13e65fa3aec8536102496e9412a71a69f9be6a4f Oct 30 12:52:10.502651 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:10.502568 2572 generic.go:356] "Generic (PLEG): container finished" podID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerID="9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce" exitCode=0 Oct 30 12:52:10.503073 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:10.502659 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd51364d-3838-4a3d-b470-ccf6483121e7","Type":"ContainerDied","Data":"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce"} Oct 30 12:52:10.503073 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:10.502744 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd51364d-3838-4a3d-b470-ccf6483121e7","Type":"ContainerStarted","Data":"cbad41beb16322a7113878ce13e65fa3aec8536102496e9412a71a69f9be6a4f"} Oct 30 12:52:13.447753 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:13.447723 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:13.518563 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:13.518523 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd51364d-3838-4a3d-b470-ccf6483121e7","Type":"ContainerStarted","Data":"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de"} Oct 30 12:52:13.518697 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:13.518617 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd51364d-3838-4a3d-b470-ccf6483121e7","Type":"ContainerStarted","Data":"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a"} Oct 30 12:52:14.525974 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:14.525938 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd51364d-3838-4a3d-b470-ccf6483121e7","Type":"ContainerStarted","Data":"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b"} Oct 30 12:52:14.526362 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:14.526001 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd51364d-3838-4a3d-b470-ccf6483121e7","Type":"ContainerStarted","Data":"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde"} Oct 30 12:52:14.526362 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:14.526028 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd51364d-3838-4a3d-b470-ccf6483121e7","Type":"ContainerStarted","Data":"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94"} Oct 30 12:52:14.526362 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:14.526051 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd51364d-3838-4a3d-b470-ccf6483121e7","Type":"ContainerStarted","Data":"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12"} Oct 30 12:52:14.598705 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:14.598650 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.755956898 podStartE2EDuration="5.598636834s" totalImagesPullingTime="2.826836456s" totalInitContainerRuntime="15.84348ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:14.597857665 +0000 UTC m=+62.984811563" watchObservedRunningTime="2026-10-30 12:52:14.598636834 +0000 UTC m=+62.985590731" Oct 30 12:52:14.835973 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:14.835896 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:15.509952 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:15.509923 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:17.909710 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:17.909671 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:52:17.909710 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:17.909714 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs\") pod \"network-metrics-daemon-b68v5\" (UID: \"050eebb4-7228-4189-b9ec-5bac3d7ed58e\") " pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:52:17.922938 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:17.922912 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 12:52:17.922938 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:17.922911 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 12:52:17.932345 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:17.932308 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs\") pod \"network-metrics-daemon-b68v5\" (UID: \"050eebb4-7228-4189-b9ec-5bac3d7ed58e\") " pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:52:17.932463 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:17.932431 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:52:17.952243 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:17.952213 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n54s8\"" Oct 30 12:52:17.960301 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:17.960277 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:52:18.010457 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:18.010422 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4qf\" (UniqueName: \"kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf\") pod \"network-check-target-dgnrj\" (UID: \"2915194d-a9f7-460c-aee2-af5262da4af4\") " pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:52:18.023627 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:18.023592 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 12:52:18.043667 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:18.043636 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 12:52:18.054156 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:18.054112 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv4qf\" (UniqueName: \"kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf\") pod \"network-check-target-dgnrj\" (UID: \"2915194d-a9f7-460c-aee2-af5262da4af4\") " pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:52:18.234511 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:18.234413 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sb6s2\"" Oct 30 12:52:18.241582 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:18.241549 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:52:18.243319 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:18.243301 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rnjq7\"" Oct 30 12:52:18.251616 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:18.251596 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:52:18.259795 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:18.259771 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b68v5"] Oct 30 12:52:18.266039 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:52:18.266008 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050eebb4_7228_4189_b9ec_5bac3d7ed58e.slice/crio-01b39fe67c13da38f480a2f3596be2da5730a4eac612b80e44670dd6127c08c6 WatchSource:0}: Error finding container 01b39fe67c13da38f480a2f3596be2da5730a4eac612b80e44670dd6127c08c6: Status 404 returned error can't find the container with id 01b39fe67c13da38f480a2f3596be2da5730a4eac612b80e44670dd6127c08c6 Oct 30 12:52:18.543320 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:18.543233 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b68v5" event={"ID":"050eebb4-7228-4189-b9ec-5bac3d7ed58e","Type":"ContainerStarted","Data":"01b39fe67c13da38f480a2f3596be2da5730a4eac612b80e44670dd6127c08c6"} Oct 30 12:52:18.547510 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:18.547465 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dgnrj"] Oct 30 12:52:18.558125 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:52:18.558096 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2915194d_a9f7_460c_aee2_af5262da4af4.slice/crio-1a261f8f4d5b048da48aa0ba4f41bfa16d0d1540b67f421a7e6e9a95ac429c07 WatchSource:0}: Error finding container 1a261f8f4d5b048da48aa0ba4f41bfa16d0d1540b67f421a7e6e9a95ac429c07: Status 404 returned error can't find the container with id 1a261f8f4d5b048da48aa0ba4f41bfa16d0d1540b67f421a7e6e9a95ac429c07 Oct 30 12:52:18.563776 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:18.563748 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2xg75"] Oct 30 12:52:18.567799 ip-10-0-138-103 kubenswrapper[2572]: W1030 12:52:18.567773 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b497ad2_5bd4_4b7b_881d_d94c15fda49c.slice/crio-7db93a08012b3adf1c4fd552b3daef9c4f91595155f5f1f887c23ef6f3400c38 WatchSource:0}: Error finding container 7db93a08012b3adf1c4fd552b3daef9c4f91595155f5f1f887c23ef6f3400c38: Status 404 returned error can't find the container with id 7db93a08012b3adf1c4fd552b3daef9c4f91595155f5f1f887c23ef6f3400c38 Oct 30 12:52:19.550294 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:19.550205 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2xg75" event={"ID":"0b497ad2-5bd4-4b7b-881d-d94c15fda49c","Type":"ContainerStarted","Data":"7db93a08012b3adf1c4fd552b3daef9c4f91595155f5f1f887c23ef6f3400c38"} Oct 30 12:52:19.552647 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:19.552613 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b68v5" event={"ID":"050eebb4-7228-4189-b9ec-5bac3d7ed58e","Type":"ContainerStarted","Data":"6771a5c14fa2eebd5121b78633728ee2a040f40fbac0cf8ccfa16385a5ef934b"} Oct 30 12:52:19.552761 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:19.552674 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b68v5" event={"ID":"050eebb4-7228-4189-b9ec-5bac3d7ed58e","Type":"ContainerStarted","Data":"20f4eead70c533cbec06b1ae729cf9eed9d1605b43054c9f05c1a8b74d791273"} Oct 30 12:52:19.556122 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:19.556095 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dgnrj" event={"ID":"2915194d-a9f7-460c-aee2-af5262da4af4","Type":"ContainerStarted","Data":"1a261f8f4d5b048da48aa0ba4f41bfa16d0d1540b67f421a7e6e9a95ac429c07"} Oct 30 12:52:19.606076 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:19.606020 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-b68v5" podStartSLOduration=66.680006443 podStartE2EDuration="1m7.606004439s" totalImagesPullingTime="925.997996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:19.602902728 +0000 UTC m=+67.989856626" watchObservedRunningTime="2026-10-30 12:52:19.606004439 +0000 UTC m=+67.992958335" Oct 30 12:52:22.570949 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:22.570909 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dgnrj" event={"ID":"2915194d-a9f7-460c-aee2-af5262da4af4","Type":"ContainerStarted","Data":"eee7ff35b38e20c47413b23de051c10b12200d97dc7094f2b4f18950b72afeb3"} Oct 30 12:52:22.571414 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:22.571098 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:52:22.613156 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:22.613091 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dgnrj" podStartSLOduration=67.58165603 podStartE2EDuration="1m10.613072323s" totalImagesPullingTime="3.031416293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:22.610834145 +0000 UTC m=+70.997788053" watchObservedRunningTime="2026-10-30 12:52:22.613072323 +0000 UTC m=+71.000026220" Oct 30 12:52:23.305083 ip-10-0-138-103 kubenswrapper[2572]: I1030 12:52:23.304899 2572 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:52:23.305098 ip-10-0-138-103 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 12:52:23.318846 ip-10-0-138-103 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 12:52:23.319027 ip-10-0-138-103 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 12:52:23.319080 ip-10-0-138-103 systemd[1]: kubelet.service: Consumed 4.532s CPU time. Oct 30 12:52:23.335240 ip-10-0-138-103 systemd[1]: Starting Kubernetes Kubelet... Oct 30 12:52:23.398711 ip-10-0-138-103 kubenswrapper[7010]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:52:23.398711 ip-10-0-138-103 kubenswrapper[7010]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 12:52:23.398711 ip-10-0-138-103 kubenswrapper[7010]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:52:23.398711 ip-10-0-138-103 kubenswrapper[7010]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 12:52:23.400759 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400670 7010 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:52:23.400759 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400758 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:52:23.400759 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400762 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400767 7010 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400770 7010 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400774 7010 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400777 7010 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400780 7010 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400782 7010 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400785 7010 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400788 7010 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400791 7010 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400793 7010 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400796 7010 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400798 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400801 7010 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400804 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400807 7010 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400809 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400812 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400821 7010 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400824 7010 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:52:23.400863 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400827 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400830 7010 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400833 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400836 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400838 7010 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400841 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400843 7010 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400846 7010 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400849 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400852 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400855 7010 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400858 7010 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400860 7010 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400863 7010 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400867 7010 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400871 7010 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400876 7010 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400879 7010 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400882 7010 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:52:23.401337 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400885 7010 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400888 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400890 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400893 7010 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400896 7010 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400900 7010 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400904 7010 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400907 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400911 7010 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400914 7010 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400916 7010 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400920 7010 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400923 7010 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400926 7010 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400928 7010 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400931 7010 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400933 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400936 7010 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400939 7010 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:52:23.401844 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400941 7010 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400944 7010 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400946 7010 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400949 7010 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400952 7010 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400954 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400957 7010 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400959 7010 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400962 7010 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400964 7010 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400967 7010 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400969 7010 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400972 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400975 7010 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400978 7010 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400980 7010 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400983 7010 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400985 7010 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400989 7010 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400991 7010 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:52:23.402304 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400994 7010 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400997 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.400999 7010 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401002 7010 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401004 7010 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401007 7010 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401009 7010 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401012 7010 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401014 7010 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401017 7010 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401019 7010 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401022 7010 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401024 7010 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401027 7010 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401029 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401032 7010 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401034 7010 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401037 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401039 7010 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401042 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:52:23.402831 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401045 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401047 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401050 7010 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401052 7010 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401055 7010 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401058 7010 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401905 7010 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401913 7010 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401915 7010 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401918 7010 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401922 7010 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401924 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401927 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401930 7010 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401932 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401935 7010 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401938 7010 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401940 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401943 7010 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:52:23.403341 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401946 7010 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401949 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401951 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401954 7010 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401957 7010 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401959 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401962 7010 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401965 7010 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401968 7010 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401971 7010 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401974 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401976 7010 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401979 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401982 7010 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401984 7010 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401987 7010 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401990 7010 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401992 7010 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.401995 7010 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:52:23.403828 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402003 7010 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402006 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402009 7010 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402012 7010 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402014 7010 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402017 7010 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402019 7010 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402023 7010 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402025 7010 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402028 7010 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402030 7010 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402033 7010 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402036 7010 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402039 7010 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402041 7010 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402044 7010 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402047 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402050 7010 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402053 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402055 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:52:23.404315 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402058 7010 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402061 7010 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402063 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402066 7010 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402068 7010 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402071 7010 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402073 7010 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402076 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402079 7010 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402081 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402084 7010 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402086 7010 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402089 7010 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402097 7010 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402100 7010 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402102 7010 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402105 7010 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402107 7010 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402110 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:52:23.404794 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402112 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402115 7010 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402117 7010 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402120 7010 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402122 7010 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402125 7010 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402128 7010 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402131 7010 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402134 7010 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402136 7010 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402139 7010 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402142 7010 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402144 7010 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402147 7010 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402150 7010 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402152 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402162 7010 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402165 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402167 7010 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402170 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:52:23.405259 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402172 7010 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402175 7010 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402178 7010 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402180 7010 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402184 7010 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402188 7010 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402191 7010 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402200 7010 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402203 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402206 7010 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402208 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402211 7010 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402213 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402216 7010 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402220 7010 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402801 7010 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402807 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402810 7010 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402816 7010 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:52:23.405774 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402819 7010 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402822 7010 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402824 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402827 7010 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402829 7010 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402832 7010 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402835 7010 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402838 7010 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402841 7010 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402844 7010 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402847 7010 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402849 7010 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402852 7010 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402855 7010 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402858 7010 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402860 7010 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402863 7010 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402865 7010 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402868 7010 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402871 7010 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:52:23.406251 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402881 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402884 7010 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402886 7010 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402889 7010 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402892 7010 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402894 7010 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402897 7010 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402899 7010 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402902 7010 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402905 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402907 7010 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402911 7010 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402914 7010 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402917 7010 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402919 7010 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402922 7010 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402924 7010 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402927 7010 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402930 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402933 7010 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:52:23.406752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402936 7010 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402939 7010 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402942 7010 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402945 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402947 7010 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402950 7010 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402952 7010 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402955 7010 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402958 7010 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402960 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402963 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402965 7010 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402968 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402976 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402979 7010 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402982 7010 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402984 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402987 7010 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402989 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:52:23.407260 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402992 7010 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402994 7010 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402997 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.402999 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403003 7010 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403006 7010 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403008 7010 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403011 7010 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403013 7010 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403016 7010 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403018 7010 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403021 7010 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403025 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403028 7010 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403030 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403033 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403035 7010 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403038 7010 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403041 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403043 7010 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:52:23.407738 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403046 7010 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403048 7010 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403051 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403053 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403056 7010 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403059 7010 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403061 7010 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403070 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403073 7010 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403075 7010 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403078 7010 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403080 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403083 7010 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403085 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403089 7010 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403093 7010 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403097 7010 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403100 7010 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403102 7010 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:52:23.408216 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403105 7010 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403107 7010 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403112 7010 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403115 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403216 7010 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403225 7010 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403239 7010 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403243 7010 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403248 7010 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403251 7010 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403256 7010 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403260 7010 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403264 7010 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403267 7010 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403270 7010 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403274 7010 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403277 7010 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403280 7010 flags.go:64] FLAG: --cgroup-root="" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403289 7010 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403293 7010 flags.go:64] FLAG: --client-ca-file="" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403296 7010 flags.go:64] FLAG: --cloud-provider="external" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403306 7010 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403312 7010 flags.go:64] FLAG: --cluster-domain="" Oct 30 12:52:23.408974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403315 7010 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403318 7010 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403321 7010 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403325 7010 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403329 7010 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403332 7010 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403337 7010 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403340 7010 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403343 7010 flags.go:64] FLAG: --contention-profiling="false" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403346 7010 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403349 7010 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403353 7010 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403356 7010 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403360 7010 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403367 7010 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403370 7010 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403373 7010 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403376 7010 flags.go:64] FLAG: --enable-server="true" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403379 7010 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403386 7010 flags.go:64] FLAG: --event-burst="100" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403389 7010 flags.go:64] FLAG: --event-qps="50" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403392 7010 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403395 7010 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403398 7010 flags.go:64] FLAG: --eviction-hard="" Oct 30 12:52:23.409687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403403 7010 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403406 7010 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403410 7010 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403413 7010 flags.go:64] FLAG: --eviction-soft="" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403416 7010 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403419 7010 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403421 7010 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403432 7010 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403435 7010 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403438 7010 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403441 7010 flags.go:64] FLAG: --feature-gates="" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403445 7010 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403448 7010 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403451 7010 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403455 7010 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403458 7010 flags.go:64] FLAG: --healthz-port="10248" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403461 7010 flags.go:64] FLAG: --help="false" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403464 7010 flags.go:64] FLAG: --hostname-override="ip-10-0-138-103.ec2.internal" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403467 7010 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403470 7010 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403473 7010 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403477 7010 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403481 7010 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403497 7010 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 12:52:23.410307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403500 7010 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403503 7010 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403506 7010 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403509 7010 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403512 7010 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403515 7010 flags.go:64] FLAG: --kube-reserved="" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403518 7010 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403521 7010 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403524 7010 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403527 7010 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403530 7010 flags.go:64] FLAG: --lock-file="" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403533 7010 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403536 7010 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403539 7010 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403544 7010 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403547 7010 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403557 7010 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403560 7010 flags.go:64] FLAG: --logging-format="text" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403563 7010 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403566 7010 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403569 7010 flags.go:64] FLAG: --manifest-url="" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403572 7010 flags.go:64] FLAG: --manifest-url-header="" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403577 7010 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403580 7010 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403583 7010 flags.go:64] FLAG: --max-pods="110" Oct 30 12:52:23.410906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403586 7010 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403589 7010 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403592 7010 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403595 7010 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403598 7010 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403602 7010 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403605 7010 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403612 7010 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403616 7010 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403618 7010 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403622 7010 flags.go:64] FLAG: --pod-cidr="" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403624 7010 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403627 7010 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403630 7010 flags.go:64] FLAG: --pods-per-core="0" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403633 7010 flags.go:64] FLAG: --port="10250" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403637 7010 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403640 7010 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0adfb01818ff2b9e7" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403643 7010 flags.go:64] FLAG: --qos-reserved="" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403646 7010 flags.go:64] FLAG: --read-only-port="10255" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403649 7010 flags.go:64] FLAG: --register-node="true" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403652 7010 flags.go:64] FLAG: --register-with-taints="" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403656 7010 flags.go:64] FLAG: --registry-burst="10" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403659 7010 flags.go:64] FLAG: --registry-qps="5" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403662 7010 flags.go:64] FLAG: --reserved-cpus="" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403671 7010 flags.go:64] FLAG: --reserved-memory="" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403675 7010 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 12:52:23.411511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403678 7010 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403681 7010 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403684 7010 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403686 7010 flags.go:64] FLAG: --runonce="false" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403689 7010 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403692 7010 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403695 7010 flags.go:64] FLAG: --seccomp-default="false" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403698 7010 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403701 7010 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403704 7010 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403713 7010 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403718 7010 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403721 7010 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403724 7010 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403727 7010 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403729 7010 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403732 7010 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403735 7010 flags.go:64] FLAG: --system-cgroups="" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403738 7010 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403743 7010 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403746 7010 flags.go:64] FLAG: --tls-cert-file="" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403749 7010 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403756 7010 flags.go:64] FLAG: --tls-min-version="" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403759 7010 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403761 7010 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 12:52:23.412141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403764 7010 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403767 7010 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403770 7010 flags.go:64] FLAG: --v="2" Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403775 7010 flags.go:64] FLAG: --version="false" Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403779 7010 flags.go:64] FLAG: --vmodule="" Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403782 7010 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.403791 7010 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403960 7010 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403964 7010 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403966 7010 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403969 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403972 7010 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403974 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403977 7010 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403979 7010 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403982 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403986 7010 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403989 7010 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403993 7010 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403995 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.403998 7010 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:52:23.412760 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404000 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404003 7010 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404005 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404008 7010 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404011 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404013 7010 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404016 7010 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404019 7010 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404021 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404024 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404026 7010 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404029 7010 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404032 7010 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404034 7010 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404037 7010 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404039 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404042 7010 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404044 7010 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404047 7010 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404050 7010 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:52:23.413265 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404052 7010 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404055 7010 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404058 7010 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404060 7010 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404064 7010 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404068 7010 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404072 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404077 7010 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404080 7010 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404083 7010 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404086 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404089 7010 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404091 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404094 7010 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404097 7010 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404099 7010 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404102 7010 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404104 7010 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404107 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404109 7010 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:52:23.413800 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404112 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404115 7010 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404117 7010 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404120 7010 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404122 7010 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404125 7010 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404127 7010 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404130 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404133 7010 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404135 7010 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404138 7010 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404141 7010 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404143 7010 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404146 7010 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404148 7010 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404151 7010 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404153 7010 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404156 7010 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404158 7010 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404162 7010 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:52:23.414390 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404165 7010 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404168 7010 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404170 7010 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404173 7010 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404175 7010 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404178 7010 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404181 7010 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404183 7010 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404187 7010 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404191 7010 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404194 7010 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404198 7010 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404201 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404203 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404206 7010 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404209 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404211 7010 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404214 7010 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404217 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:52:23.415137 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404219 7010 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:52:23.415654 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404221 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:52:23.415654 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404225 7010 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:52:23.415654 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404227 7010 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:52:23.415654 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404230 7010 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:52:23.415654 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404233 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:52:23.415654 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404235 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:52:23.415654 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404238 7010 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:52:23.415654 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404241 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:52:23.415654 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404243 7010 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:52:23.415654 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404246 7010 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:52:23.415654 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404248 7010 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:52:23.415654 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404253 7010 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:52:23.415654 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.404278 7010 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:52:23.415654 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404314 7010 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:52:23.415654 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404320 7010 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:52:23.415654 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404323 7010 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404326 7010 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404330 7010 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404334 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404337 7010 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404340 7010 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404342 7010 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404345 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404348 7010 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404350 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404353 7010 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404356 7010 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404358 7010 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404361 7010 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404363 7010 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404366 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404368 7010 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404371 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404373 7010 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:52:23.416046 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404376 7010 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404378 7010 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404381 7010 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404384 7010 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404387 7010 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404389 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404393 7010 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404396 7010 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404398 7010 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404401 7010 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404405 7010 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404408 7010 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404410 7010 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404413 7010 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404416 7010 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404418 7010 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404421 7010 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404423 7010 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404426 7010 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404429 7010 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:52:23.416536 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404431 7010 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404434 7010 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404436 7010 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404439 7010 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404442 7010 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404444 7010 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404449 7010 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404452 7010 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404456 7010 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404458 7010 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404461 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404464 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404467 7010 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404470 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404473 7010 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404476 7010 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404479 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404482 7010 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404497 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:52:23.417174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404500 7010 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404503 7010 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404505 7010 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404509 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404512 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404515 7010 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404518 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404520 7010 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404523 7010 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404525 7010 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404528 7010 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404531 7010 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404533 7010 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404536 7010 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404538 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404541 7010 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404544 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404546 7010 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404549 7010 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:52:23.417697 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404552 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404555 7010 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404557 7010 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404560 7010 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404562 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404565 7010 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404567 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404570 7010 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404572 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404575 7010 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404578 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404581 7010 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404585 7010 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404587 7010 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404590 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404592 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404596 7010 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404600 7010 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404602 7010 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404605 7010 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:52:23.418143 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404608 7010 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404610 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404613 7010 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404616 7010 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404618 7010 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404621 7010 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.404623 7010 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.408849 7010 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.408871 7010 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.408895 7010 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.408902 7010 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409066 7010 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409073 7010 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409078 7010 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409083 7010 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409088 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409092 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409096 7010 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409101 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:52:23.418653 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409105 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409110 7010 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409114 7010 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409123 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409129 7010 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409133 7010 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409137 7010 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409143 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409147 7010 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409152 7010 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409157 7010 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409161 7010 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409166 7010 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409171 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409175 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409185 7010 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409190 7010 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409194 7010 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409198 7010 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409203 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:52:23.419099 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409207 7010 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409212 7010 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409216 7010 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409220 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409224 7010 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409229 7010 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409233 7010 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409237 7010 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409246 7010 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409250 7010 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409274 7010 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409299 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409306 7010 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409311 7010 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409319 7010 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409326 7010 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409329 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409332 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409335 7010 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409340 7010 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:52:23.419612 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409343 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409346 7010 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409350 7010 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409353 7010 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409364 7010 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409465 7010 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409469 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409473 7010 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409476 7010 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409479 7010 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409496 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409499 7010 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409503 7010 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409505 7010 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409508 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409512 7010 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409514 7010 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409517 7010 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409520 7010 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409523 7010 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:52:23.420093 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409525 7010 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409528 7010 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409531 7010 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409534 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409537 7010 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409539 7010 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409543 7010 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409546 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409549 7010 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409551 7010 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409555 7010 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409558 7010 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409561 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409567 7010 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409572 7010 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409575 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409577 7010 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409580 7010 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409583 7010 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:52:23.420599 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409586 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409589 7010 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409591 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409600 7010 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409603 7010 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409605 7010 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409608 7010 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409611 7010 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409615 7010 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409619 7010 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409622 7010 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409625 7010 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409628 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409631 7010 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409633 7010 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409636 7010 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409639 7010 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409642 7010 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:52:23.421092 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409646 7010 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:52:23.421539 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.409682 7010 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:52:23.421539 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409827 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:52:23.421539 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409833 7010 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:52:23.421539 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409836 7010 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:52:23.421539 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409839 7010 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:52:23.421539 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409841 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:52:23.421539 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409844 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:52:23.421539 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409847 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:52:23.421539 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409850 7010 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:52:23.421539 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409852 7010 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:52:23.421539 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409855 7010 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:52:23.421539 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409857 7010 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:52:23.421539 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409860 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:52:23.421539 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409862 7010 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:52:23.421539 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409867 7010 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409871 7010 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409874 7010 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409878 7010 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409880 7010 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409883 7010 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409886 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409888 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409891 7010 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409894 7010 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409896 7010 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409899 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409901 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409904 7010 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409906 7010 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409909 7010 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409912 7010 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409914 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409917 7010 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:52:23.421905 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409919 7010 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409922 7010 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409924 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409927 7010 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409930 7010 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409932 7010 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409936 7010 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409938 7010 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409941 7010 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409943 7010 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409946 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409948 7010 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409951 7010 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409954 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409956 7010 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409959 7010 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409961 7010 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409964 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409967 7010 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409969 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:52:23.422362 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409972 7010 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409974 7010 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409977 7010 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409979 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409982 7010 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409984 7010 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409987 7010 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409989 7010 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409992 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409994 7010 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.409997 7010 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410000 7010 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410002 7010 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410005 7010 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410007 7010 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410010 7010 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410012 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410015 7010 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410018 7010 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410021 7010 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:52:23.422846 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410024 7010 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410027 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410029 7010 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410032 7010 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410034 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410037 7010 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410039 7010 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410042 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410045 7010 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410048 7010 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410050 7010 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410053 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410055 7010 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410058 7010 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410061 7010 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410063 7010 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410066 7010 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410070 7010 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410073 7010 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:52:23.423373 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410076 7010 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:52:23.423994 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410080 7010 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:52:23.423994 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410082 7010 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:52:23.423994 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410086 7010 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:52:23.423994 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410089 7010 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:52:23.423994 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410091 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:52:23.423994 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410094 7010 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:52:23.423994 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410096 7010 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:52:23.423994 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410099 7010 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:52:23.423994 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410101 7010 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:52:23.423994 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410104 7010 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:52:23.423994 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410107 7010 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:52:23.423994 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410109 7010 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:52:23.423994 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410113 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:52:23.423994 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410116 7010 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:52:23.423994 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.410139 7010 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 12:52:23.423994 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410170 7010 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410174 7010 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410177 7010 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410181 7010 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410184 7010 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410187 7010 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410190 7010 feature_gate.go:386] unrecognized feature gate: Example Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410193 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410196 7010 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410199 7010 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410202 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410204 7010 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410207 7010 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410209 7010 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410212 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410215 7010 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410218 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410220 7010 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410223 7010 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410225 7010 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 12:52:23.424376 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410228 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410230 7010 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410233 7010 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410235 7010 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410238 7010 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410241 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410243 7010 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410246 7010 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410248 7010 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410251 7010 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410254 7010 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410256 7010 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410259 7010 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410261 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410266 7010 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410269 7010 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410271 7010 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410274 7010 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410277 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410280 7010 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 12:52:23.424876 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410283 7010 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410285 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410288 7010 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410290 7010 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410293 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410296 7010 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410298 7010 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410301 7010 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410304 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410306 7010 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410309 7010 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410312 7010 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410314 7010 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410317 7010 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410321 7010 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410325 7010 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410328 7010 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410330 7010 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410333 7010 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 12:52:23.425418 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410336 7010 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410338 7010 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410341 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410343 7010 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410348 7010 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410351 7010 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410353 7010 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410357 7010 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410360 7010 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410362 7010 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410365 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410369 7010 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410371 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410374 7010 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410377 7010 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410379 7010 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410382 7010 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410385 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410387 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410390 7010 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 12:52:23.426174 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410392 7010 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410395 7010 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410398 7010 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410400 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410403 7010 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410406 7010 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410410 7010 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410413 7010 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410416 7010 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410419 7010 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410421 7010 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410424 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410427 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410429 7010 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410432 7010 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410434 7010 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410437 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410440 7010 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410443 7010 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 12:52:23.426939 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410446 7010 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 12:52:23.428085 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410448 7010 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 12:52:23.428085 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410451 7010 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 12:52:23.428085 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410453 7010 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 12:52:23.428085 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410457 7010 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 12:52:23.428085 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410460 7010 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 12:52:23.428085 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410463 7010 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 12:52:23.428085 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:23.410465 7010 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 12:52:23.428085 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.410537 7010 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 12:52:23.428085 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.411583 7010 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 12:52:23.428085 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.411646 7010 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 12:52:23.428085 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.411821 7010 server.go:1014] "Starting client certificate rotation" Oct 30 12:52:23.428085 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.411961 7010 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 12:52:23.428085 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.412016 7010 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:46:12 +0000 UTC" deadline="2028-07-30 05:23:32.819402474 +0000 UTC" Oct 30 12:52:23.428085 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.412043 7010 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15328h31m9.407361479s" Oct 30 12:52:23.428085 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.412275 7010 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:52:23.428762 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.413337 7010 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 12:52:23.428762 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.414524 7010 log.go:25] "Validated CRI v1 runtime API" Oct 30 12:52:23.428762 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.416414 7010 log.go:25] "Validated CRI v1 image API" Oct 30 12:52:23.428762 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.417086 7010 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 12:52:23.428762 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.419389 7010 fs.go:135] Filesystem UUIDs: map[540a3b59-f35a-45d8-bac1-e36ed708e86c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9084f533-0f76-4389-9afc-3b50c9997d9b:/dev/nvme0n1p3] Oct 30 12:52:23.429023 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.419410 7010 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01b39fe67c13da38f480a2f3596be2da5730a4eac612b80e44670dd6127c08c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01b39fe67c13da38f480a2f3596be2da5730a4eac612b80e44670dd6127c08c6/userdata/shm major:0 minor:475 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11dbf715ab7856ce862a47a1fee18a01aa768b32f7005a999e8af99b8df53c82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11dbf715ab7856ce862a47a1fee18a01aa768b32f7005a999e8af99b8df53c82/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/153896f3b4cb172c890a65850fdd9f5159745eac48318cf3426b69f348ecc283/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/153896f3b4cb172c890a65850fdd9f5159745eac48318cf3426b69f348ecc283/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a261f8f4d5b048da48aa0ba4f41bfa16d0d1540b67f421a7e6e9a95ac429c07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a261f8f4d5b048da48aa0ba4f41bfa16d0d1540b67f421a7e6e9a95ac429c07/userdata/shm major:0 minor:478 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2159f410d7248ba49f61a9f95c4b8a0a641d6137854f345887de4ed27c15e497/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2159f410d7248ba49f61a9f95c4b8a0a641d6137854f345887de4ed27c15e497/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29ccd7fa19dcd4c1dfac94a30a27d98cce2a48b7f3a8782fc46acb5e8f2994a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29ccd7fa19dcd4c1dfac94a30a27d98cce2a48b7f3a8782fc46acb5e8f2994a8/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c8c1de77cf6c4cbda32164f7696a993e56def4e4db1de3840cb5ab2f7663344/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c8c1de77cf6c4cbda32164f7696a993e56def4e4db1de3840cb5ab2f7663344/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b50ebe230b9d7b6f04e8d6ea27abcbb07763957be846c73dbf54938777b177c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b50ebe230b9d7b6f04e8d6ea27abcbb07763957be846c73dbf54938777b177c/userdata/shm major:0 minor:360 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a3430d1e2186a2de5e9637a00a4c6d50862f1804bfb8fed6b6086c0a2cf528c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a3430d1e2186a2de5e9637a00a4c6d50862f1804bfb8fed6b6086c0a2cf528c/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d83332a9c55d3325c784e5856c29be7531d22e2c24b35dd533ed8cfd1711093/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d83332a9c55d3325c784e5856c29be7531d22e2c24b35dd533ed8cfd1711093/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7db93a08012b3adf1c4fd552b3daef9c4f91595155f5f1f887c23ef6f3400c38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7db93a08012b3adf1c4fd552b3daef9c4f91595155f5f1f887c23ef6f3400c38/userdata/shm major:0 minor:480 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87ba1850a1899bb73ae42d0233d441fb994378aeee19d0c4cec52bad58abf701/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87ba1850a1899bb73ae42d0233d441fb994378aeee19d0c4cec52bad58abf701/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91404ee3622320376e4465fbd87101539743f97d8873143673c38ae1a64368fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91404ee3622320376e4465fbd87101539743f97d8873143673c38ae1a64368fe/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98c3ddec071d956e18ff654cccc97b8264694b1c702c2c161017e23ea91bfd27/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98c3ddec071d956e18ff654cccc97b8264694b1c702c2c161017e23ea91bfd27/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b05c2ce173dda96624c02c816fa0f7b5b5baeca0930eb0255007fd6da3fe6dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b05c2ce173dda96624c02c816fa0f7b5b5baeca0930eb0255007fd6da3fe6dd/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbad41beb16322a7113878ce13e65fa3aec8536102496e9412a71a69f9be6a4f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbad41beb16322a7113878ce13e65fa3aec8536102496e9412a71a69f9be6a4f/userdata/shm major:0 minor:438 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd9622a5d40c0137c96cff0882970b2c6cb0a143371074a75280c334383125fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd9622a5d40c0137c96cff0882970b2c6cb0a143371074a75280c334383125fd/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cece46519ef9d1fce8d5bb86eafa8086a24c14bf3a983d78738900b8eace0b0f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cece46519ef9d1fce8d5bb86eafa8086a24c14bf3a983d78738900b8eace0b0f/userdata/shm major:0 minor:252 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d751490f72a625e19a73650397f038836285aff2f8e41b56ec64e5b3b80f7ec7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d751490f72a625e19a73650397f038836285aff2f8e41b56ec64e5b3b80f7ec7/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1c4b32f926d9b2b17727f862be9d30fb04dd7f0de66dcefadf536d2c3a87092/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1c4b32f926d9b2b17727f862be9d30fb04dd7f0de66dcefadf536d2c3a87092/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8176bf5fb9c579053d7eacdc09d0fcc59da745c7adc6c811ae87881f314db32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8176bf5fb9c579053d7eacdc09d0fcc59da745c7adc6c811ae87881f314db32/userdata/shm major:0 minor:250 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9f1e9c6bb635bc9663b6fad1d374d211b92838f8bcf96afa913ecde4e5147e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9f1e9c6bb635bc9663b6fad1d374d211b92838f8bcf96afa913ecde4e5147e0/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ecf7cf31b72f03a1a62c9eb61c6584dd6751f35a49c9bf5e44d9b9433ee9f31f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ecf7cf31b72f03a1a62c9eb61c6584dd6751f35a49c9bf5e44d9b9433ee9f31f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f142aab910f0986998f5a4fdb8b0bc68f393b6aaee1b89d1cbe79f308e2195af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f142aab910f0986998f5a4fdb8b0bc68f393b6aaee1b89d1cbe79f308e2195af/userdata/shm major:0 minor:361 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/050eebb4-7228-4189-b9ec-5bac3d7ed58e/volumes/kubernetes.io~projected/kube-api-access-x6ctl:{mountpoint:/var/lib/kubelet/pods/050eebb4-7228-4189-b9ec-5bac3d7ed58e/volumes/kubernetes.io~projected/kube-api-access-x6ctl major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/050eebb4-7228-4189-b9ec-5bac3d7ed58e/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/050eebb4-7228-4189-b9ec-5bac3d7ed58e/volumes/kubernetes.io~secret/metrics-certs major:0 minor:470 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09b6fe90-3962-49ef-bc16-486d09469b22/volumes/kubernetes.io~projected/kube-api-access-mrkbx:{mountpoint:/var/lib/kubelet/pods/09b6fe90-3962-49ef-bc16-486d09469b22/volumes/kubernetes.io~projected/kube-api-access-mrkbx major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09b6fe90-3962-49ef-bc16-486d09469b22/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/09b6fe90-3962-49ef-bc16-486d09469b22/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b497ad2-5bd4-4b7b-881d-d94c15fda49c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/0b497ad2-5bd4-4b7b-881d-d94c15fda49c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:474 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1069a5ce-5720-4e0d-8bb8-cc874020e3dd/volumes/kubernetes.io~projected/kube-api-access-wdpc7:{mountpoint:/var/lib/kubelet/pods/1069a5ce-5720-4e0d-8bb8-cc874020e3dd/volumes/kubernetes.io~projected/kube-api-access-wdpc7 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1069a5ce-5720-4e0d-8bb8-cc874020e3dd/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/1069a5ce-5720-4e0d-8bb8-cc874020e3dd/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/109981ff-df1e-4279-bc7a-1e2b09f30ad1/volumes/kubernetes.io~projected/kube-api-access-2gqh8:{mountpoint:/var/lib/kubelet/pods/109981ff-df1e-4279-bc7a-1e2b09f30ad1/volumes/kubernetes.io~projected/kube-api-access-2gqh8 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2915194d-a9f7-460c-aee2-af5262da4af4/volumes/kubernetes.io~projected/kube-api-access-fv4qf:{mountpoint:/var/lib/kubelet/pods/2915194d-a9f7-460c-aee2-af5262da4af4/volumes/kubernetes.io~projected/kube-api-access-fv4qf major:0 minor:477 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volume-subpaths/web-config/alertmanager/9 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volume-subpaths/web-config/config-reloader/6 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~empty-dir/config-out major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~projected/kube-api-access-h2wdq:{mountpoint:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~projected/kube-api-access-h2wdq major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~projected/tls-assets major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/config-volume major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/web-config major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd/volumes/kubernetes.io~projected/kube-api-access-5kwws:{mountpoint:/var/lib/kubelet/pods/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd/volumes/kubernetes.io~projected/kube-api-access-5kwws major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31e0a09f-ff45-482d-afec-c57bd2d41cb4/volumes/kubernetes.io~projected/kube-api-access-9xdcg:{mountpoint:/var/lib/kubelet/pods/31e0a09f-ff45-482d-afec-c57bd2d41cb4/volumes/kubernetes.io~projected/kube-api-access-9xdcg major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31e0a09f-ff45-482d-afec-c57bd2d41cb4/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/31e0a09f-ff45-482d-afec-c57bd2d41cb4/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31e0a09f-ff45-482d-afec-c57bd2d41cb4/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/31e0a09f-ff45-482d-afec-c57bd2d41cb4/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/404bb84d-f044-488e-a221-389f71e61181/volumes/kubernetes.io~projected/kube-api-access-hd6nk:{mountpoint:/var/lib/kubelet/pods/404bb84d-f044-488e-a221-389f71e61181/volumes/kubernetes.io~projected/kube-api-access-hd6nk major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b885e3c-593b-45d1-b4d4-4604ca511662/volumes/kubernetes.io~projected/kube-api-access-flgvz:{mountpoint:/var/lib/kubelet/pods/5b885e3c-593b-45d1-b4d4-4604ca511662/volumes/kubernetes.io~projected/kube-api-access-flgvz major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b885e3c-593b-45d1-b4d4-4604ca511662/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/5b885e3c-593b-45d1-b4d4-4604ca511662/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b885e3c-593b-45d1-b4d4-4604ca511662/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/5b885e3c-593b-45d1-b4d4-4604ca511662/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b885e3c-593b-45d1-b4d4-4604ca511662/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/5b885e3c-593b-45d1-b4d4-4604ca511662/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60a88a61-3c25-48c8-84c5-b35bab4a058a/volumes/kubernetes.io~projected/kube-api-access-p8hp2:{mountpoint:/var/lib/kubelet/pods/60a88a61-3c25-48c8-84c5-b35bab4a058a/volumes/kubernetes.io~projected/kube-api-access-p8hp2 major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60a88a61-3c25-48c8-84c5-b35bab4a058a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/60a88a61-3c25-48c8-84c5-b35bab4a058a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60a88a61-3c25-48c8-84c5-b35bab4a058a/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/60a88a61-3c25-48c8-84c5-b35bab4a058a/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b8f3882-2505-491f-8139-1905a70a5840/volumes/kubernetes.io~projected/kube-api-access-p5bqz:{mountpoint:/var/lib/kubelet/pods/6b8f3882-2505-491f-8139-1905a70a5840/volumes/kubernetes.io~projected/kube-api-access-p5bqz major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b8f3882-2505-491f-8139-1905a70a5840/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6b8f3882-2505-491f-8139-1905a70a5840/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/741107a5-c4db-4ba9-929c-58952377929f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/741107a5-c4db-4ba9-929c-58952377929f/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83545835-dd10-4f8b-8015-25c579bdac3a/volumes/kubernetes.io~projected/kube-api-access-w5q2d:{mountpoint:/var/lib/kubelet/pods/83545835-dd10-4f8b-8015-25c579bdac3a/volumes/kubernetes.io~projected/kube-api-access-w5q2d major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~projected/kube-api-access-n2r9g:{mountpoint:/var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~projected/kube-api-access-n2r9g major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bc2e7c1-04ef-439a-90e5-57c4c200c62f/volumes/kubernetes.io~projected/kube-api-access-4dpcr:{mountpoint:/var/lib/kubelet/pods/8bc2e7c1-04ef-439a-90e5-57c4c200c62f/volumes/kubernetes.io~projected/kube-api-access-4dpcr major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9/volumes/kubernetes.io~projected/kube-api-access-x6fp9:{mountpoint:/var/lib/kubelet/pods/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9/volumes/kubernetes.io~projected/kube-api-access-x6fp9 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b663748-9129-4d2c-b900-c5ddde8b4b63/volumes/kubernetes.io~projected/kube-api-access-dvg8b:{mountpoint:/var/lib/kubelet/pods/9b663748-9129-4d2c-b900-c5ddde8b4b63/volumes/kubernetes.io~projected/kube-api-access-dvg8b major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1/volumes/kubernetes.io~projected/kube-api-access-7llpr:{mountpoint:/var/lib/kubelet/pods/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1/volumes/kubernetes.io~projected/kube-api-access-7llpr major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb39553b-ccf2-4cb0-af1c-339373ec1b13/volumes/kubernetes.io~projected/kube-api-access-bdp4l:{mountpoint:/var/lib/kubelet/pods/bb39553b-ccf2-4cb0-af1c-339373ec1b13/volumes/kubernetes.io~projected/kube-api-access-bdp4l major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb39553b-ccf2-4cb0-af1c-339373ec1b13/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/bb39553b-ccf2-4cb0-af1c-339373ec1b13/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volume-subpaths/web-config/config-reloader/2 major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volume-subpaths/web-config/prometheus/15 major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~empty-dir/config-out major:0 minor:433 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~projected/kube-api-access-k9dqr:{mountpoint:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~projected/kube-api-access-k9dqr major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~projected/tls-assets major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/config major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/web-config major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fce77b3b-df3e-4d04-945c-68dc11edc61f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/fce77b3b-df3e-4d04-945c-68dc11edc61f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fce77b3b-df3e-4d04-945c-68dc11edc61f/volumes/kubernetes.io~projected/kube-api-access-c62x8:{mountpoint:/var/lib/kubelet/pods/fce77b3b-df3e-4d04-945c-68dc11edc61f/volumes/kubernetes.io~projected/kube-api-access-c62x8 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fce77b3b-df3e-4d04-945c-68dc11edc61f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/fce77b3b-df3e-4d04-945c-68dc11edc61f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/7866c32bdfbd5e221199cfaf1d39817d169659b7abf8ef8b8fa1a04c22fb4477/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5407b2f599fd02ef305fe5bdd626af71cc1dc8cdc2a462f496e1baf918a5591b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8a6e9d9db46560ff4cd041f81f562d4e6cb4d579239d2b454264b7981fa39031/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f966ed263977410cdaba52d70e2ffcb4b95631238ad4fe173433fa1da49a6ec4/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/cd2d531615f87e9d6d1e0162ae6ddca4812e5247403ac4614abad7c6140ea39c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/60915ec6db61c15ea327dc94480913aa2c5019f997db374f70e78f534beba2de/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/4ee45749acdd57032784d632d583f331da918fa7f540ea20db8cd3b9b0aeb0e3/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/5de1f8ebd8f5d04343ee2414b55ecfa218640107ae254a748e0c0b398c5ec5a5/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f46510e6dee74736f2ea0515847e9331b20142c4ce6f19316459331552dd2289/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/b1aa9168c5eeced7bf6bbb175cec5cd612ef43212deedcf5bd06b2873907ca23/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/e06299d8ada6b6439cd848693e202f9c23be44bbab7fb96f89d3b5a5c483ba5c/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/b20427fe92f812d6c96c58a56d73e12fe593b32bf1c643d37dac54b6a1ffa884/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/37fb1c36e867c982352bb4fa561f91edf4d5a21f7b74092a679b90e031a37b27/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/cf19c025590a55002d6335785bbada322a98d73cc8292c0a668a31f17718617d/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/a23feabca079a5711f4045980a4ce54d0c73053cb6f4ab2299096485a32ed7a9/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c87f9a012e57b780326480dfeefe9b15643e9b690afbded79231e03e3a0820da/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/989ee7494440d45ba6c2b76e498f06195b6f3fab0dfb28f74a89ce70998abefd/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/543efaa479b615d4a94d7fd88a888a7d9b5d40c0f572b53d7d63fe904fd0bd39/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/97209662f2143b2919d799a7d3958fc7403dc9322cb70b96ee6dbe4a87743edc/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f153c02a960cdc2677156492e551a68e80186f585e0293081a0fcb41d01224d6/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7c99478c79ac038de2ac521160e472496f18a66f60763b9dadf6f46997c2646b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/f0410278cb2e9fca2c987eb19ee509e49702028d44d5d106461b83dc8705b477/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/b7e3f28ad5443f678d6b9ba1f51e70e1f220a7cfabea720e4566ad017b152f88/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/bffeab8f45be88e76baf9edf4b70ba795ae5375b77c80bc3498791fe28e6cc2a/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/da3165a5e16a2f9bb3901a08a643f44980e1f2b9ca1e6c36b7f75150022ec337/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/81cbd6a2d3366105544f36fedf2365fb361b04c9006cdd99ed9a6eaaeefbc902/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/42e4ddacf29c7401cdfd9aea4daefd330a92e1024c6b573863f9a3c5604d7717/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/275c4728e2cce1b1cd17f5ec7479e0ed2be99557a0911fc5e11f2bf5be6d13ed/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/529dc3acbcc714bf35c1baaedf0611a350091cefb059bb028e864907b377c03a/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/5525f24eb0abc6abad856a6d1a6d041592660a9c4846c5ef2150ffbe3100e4bb/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/c54d9758ee193e5048e8e8e6df82da4c578b3ff4aa0b0533efc9de47fde2d028/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/040b44c57c5003ea336cfd3d1050e9c4e253cdc68d5640e43b0feaf9a57cd451/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/b7e4fd7364ee421fe18ca5b16b1b68dc915aa4db0400ce1d8558c559183afc0d/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/2b25b6aef6edbdb0a0a585e2bd5e600c4ffa2aa25e39e82ba66292ddb0271854/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/013f9f063ce8df9e100eba7b1c5933368abec47b5ca37428140c454fa3d37d14/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/8b2b9372f415e3c710818190c038bc5ecc8f596ca92046cc10004fc22be09120/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/e346148203cc0527282b918aefc9d4ceec5d3f77ce71ef3e3d0754dac0a78ccf/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/4e0f8e5ce6af367b54e121a40b4b9b02a5a24e9dbaa30f3772c4f1581761ac92/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/3bb60f93f7e09aa59d14ed9bde103cc58d3e7950f0962cbeeac00e64bca94fe6/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/fe3c4f1230df8b0a3e7482791118a6504f074c5403abc086bfe15cd0f39cc99d/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/1c45b070db68d0b59049b66d89e5b11521ebc8931880072555cf8023f54f247a/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/653cdbac9c8cce1ec5cf592b754acde10ea2310611a05d024c5439a90c36515f/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/2ea66d0cce0b63e07ba56cd370503e70644756cdb2b0e87ed81e83609344ddbd/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/d0e2ac389abd5f1365fc5465e83be68559f2f849ac26348dad1bf9f400045662/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/e57dbe072eddf699b88051c181db44ff4c8c29cda0250998772a05fd0d38d2cd/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/5963fd85b460b7a5aaa5a76022a27dc797315143ac0c545f8dbc8f3f76a08584/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/318130b32a73bf09f860f7712c58e6e2ce85329e9793a215dc11a4a05d7ae3d4/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/8f6e1469ba26ba9c1e275fe3963ffe85f730a29817398b25d17e5ebe098f2132/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/d7eeff41fcb15d0626938ba04b97eb5f5156e0968563a0d93679ed818c539041/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/f38e58e76f81b1513b454ad265f18da89bafc324db2e2aa6b363ff9d1c6e0da5/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/e9c6465d753b3070531d26bd7a61b2324d79cf6486fae55de37aa3a72c0d0828/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/c2ae6e9a297a5d7fa96165a438f2bc09b66e19963e65736820aac3a5e4f6068d/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/7fecc66f4cbc57dbda95aac86da1aaf7016a492f46cc1adc37b668b779da273d/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/86f104c159e6da1d8c53ea9906235d5def7b760ab1d6afb5465ecf1c90c9db3f/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/a449b428ba0333f9f6c5260a7e82cf47c54731e7c353556d803527254411b016/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/d8d88d5e0f8aca5db7cbfc6a61dd14d7f454f282f9e67f3fbde7b5749c4be0b2/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/99285db8da2cd0ae570df2df08d2888a1e24ef61cd1b445a6b02373740891713/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/02fe0ad4a7b7096b7fd0e50002e8718daa5af572acf50cb5e4ded4b205022716/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/fdbda58951ee2c2b1689b8a804b9aeeabf2deea4ad83bf7ceeca1d364ce01e48/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/95914dd6382f96952e5b654cbecebcb2f013156fa6da404a4dee2f459323ebb3/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/d20fb289bdbbf5a487a840f186270ef04ef240743d4a66264a2cb7d07fa5c1c5/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/dfe127e2bcdcfac7e22f9e887f0c6f9e9b3c4765e04a91d69058243a0f976755/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/dee3e7df40c6f2ba533b5d7e0f833b716955d9bd54178439dfb0cfff4332e763/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/21388225156fce0cf4807520e6ffac133c12f161bef2f26e37a40cc3d69e64f2/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/b1eec55fd1c8bee922c48f5422980c20560b668a88a7549df24f82d61348c34f/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-465:{mountpoint:/var/lib/containers/storage/overlay/c65c536a8f57aa05dd2dcd1fb436f584d414b83d0e08c16bc6afd6b75e9f8343/merged major:0 minor:465 fsType:overlay blockSize:0} overlay_0-481:{mountpoint:/var/lib/containers/storage/overlay/a205301d6f1fefd2cebc1dc91d16072ab7d35bd4542eda99a138854c4e3551c8/merged major:0 minor:481 fsType:overlay blockSize:0} overlay_0-484:{mountpoint:/var/lib/containers/storage/overlay/5308f36d46dc4d12d5e64d44e8e291b1c59f6d9f71507e6b18d955fa6ee4f2a9/merged major:0 minor:484 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/4fd17148d0979b9ae295b25c8842d19d8d9895589fc94f0c09ddabfdf8c5756b/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-488:{mountpoint:/var/lib/containers/storage/overlay/4f86d82651ce19c74ab7fbe7eef0ccdf143be400c840f5eb7656c660a46ef59c/merged major:0 minor:488 fsType:overlay blockSize:0} overlay_0-490:{mountpoint:/var/lib/containers/storage/overlay/791e4c1ebe592717dec890d0c1efd37106d1045f7573af99a8473340d655805b/merged major:0 minor:490 fsType:overlay blockSize:0} overlay_0-496:{mountpoint:/var/lib/containers/storage/overlay/8c025b7630d78ccd73a48e8f3862343d29f4887e4bd402fae37ac36137b35452/merged major:0 minor:496 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/67495da3e77010c17e2401026a34ab2690979e82c6ef5704c18f066be3906ad7/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/96a3cff9ebb259bdb29bf20db101374a03cd170554989baeb377a9498a8048f7/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0a794275e9f6329a41a88f8b6b1803c95b4a623b59ae521f10d77c4cd0892651/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/34915d6a38883b2da56725f0840901f91b3426fbc93eab34a97e42dca494c81f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/d3f5fb3ab9d7cd37d73b57c9e6d195c7dc84a0f0d9ef8953b35fef68d2126f97/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/d12151c3faae0b4359acb6e1c3e3c46d9e39a567d76d8655d492ae02ab8741a1/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/2ac6d1841171fc9bc37863c41fe7a349668c89437685affba484919372e53f70/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/ad73b26c81b08fa5a80e2600704435d3262698430c763c4d2f355c4823c8eb88/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/b679244b0e28f0944d49048878e15a72656ab73f6728588b49ab3a9180e7845e/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/de98ca6a77cc241dfd83ebf3273ccbae220d151d5ce76ec4badd2d042eb06f42/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f43af86d675e035c27cd13e469835a8cefc1752d3c82f94d48549f676e3947dc/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 12:52:23.429826 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.429241 7010 manager.go:217] Machine: {Timestamp:2026-10-30 12:52:23.428382548 +0000 UTC m=+0.060569711 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095856 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21e98d4438dd25b752966da07f68c3 SystemUUID:ec21e98d-4438-dd25-b752-966da07f68c3 BootID:6a9c628e-4a2d-4cbe-a29f-63493b9a4b06 Filesystems:[{Device:/var/lib/kubelet/pods/6b8f3882-2505-491f-8139-1905a70a5840/volumes/kubernetes.io~projected/kube-api-access-p5bqz DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8176bf5fb9c579053d7eacdc09d0fcc59da745c7adc6c811ae87881f314db32/userdata/shm DeviceMajor:0 DeviceMinor:250 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:330 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2159f410d7248ba49f61a9f95c4b8a0a641d6137854f345887de4ed27c15e497/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1c4b32f926d9b2b17727f862be9d30fb04dd7f0de66dcefadf536d2c3a87092/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb39553b-ccf2-4cb0-af1c-339373ec1b13/volumes/kubernetes.io~projected/kube-api-access-bdp4l DeviceMajor:0 DeviceMinor:359 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-465 DeviceMajor:0 DeviceMinor:465 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7db93a08012b3adf1c4fd552b3daef9c4f91595155f5f1f887c23ef6f3400c38/userdata/shm DeviceMajor:0 DeviceMinor:480 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-496 DeviceMajor:0 DeviceMinor:496 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11dbf715ab7856ce862a47a1fee18a01aa768b32f7005a999e8af99b8df53c82/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cece46519ef9d1fce8d5bb86eafa8086a24c14bf3a983d78738900b8eace0b0f/userdata/shm DeviceMajor:0 DeviceMinor:252 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60a88a61-3c25-48c8-84c5-b35bab4a058a/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:239 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b885e3c-593b-45d1-b4d4-4604ca511662/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:351 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:429 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~projected/kube-api-access-k9dqr DeviceMajor:0 DeviceMinor:437 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98c3ddec071d956e18ff654cccc97b8264694b1c702c2c161017e23ea91bfd27/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:244 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-481 DeviceMajor:0 DeviceMinor:481 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-484 DeviceMajor:0 DeviceMinor:484 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-488 DeviceMajor:0 DeviceMinor:488 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1069a5ce-5720-4e0d-8bb8-cc874020e3dd/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:328 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5b885e3c-593b-45d1-b4d4-4604ca511662/volumes/kubernetes.io~projected/kube-api-access-flgvz DeviceMajor:0 DeviceMinor:358 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bc2e7c1-04ef-439a-90e5-57c4c200c62f/volumes/kubernetes.io~projected/kube-api-access-4dpcr DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1069a5ce-5720-4e0d-8bb8-cc874020e3dd/volumes/kubernetes.io~projected/kube-api-access-wdpc7 DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1/volumes/kubernetes.io~projected/kube-api-access-7llpr DeviceMajor:0 DeviceMinor:249 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:324 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:435 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-490 DeviceMajor:0 DeviceMinor:490 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd/volumes/kubernetes.io~projected/kube-api-access-5kwws DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b663748-9129-4d2c-b900-c5ddde8b4b63/volumes/kubernetes.io~projected/kube-api-access-dvg8b DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:422 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fce77b3b-df3e-4d04-945c-68dc11edc61f/volumes/kubernetes.io~projected/kube-api-access-c62x8 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb39553b-ccf2-4cb0-af1c-339373ec1b13/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:427 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbad41beb16322a7113878ce13e65fa3aec8536102496e9412a71a69f9be6a4f/userdata/shm DeviceMajor:0 DeviceMinor:438 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:422 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:422 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d83332a9c55d3325c784e5856c29be7531d22e2c24b35dd533ed8cfd1711093/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31e0a09f-ff45-482d-afec-c57bd2d41cb4/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:428 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:432 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/404bb84d-f044-488e-a221-389f71e61181/volumes/kubernetes.io~projected/kube-api-access-hd6nk DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d751490f72a625e19a73650397f038836285aff2f8e41b56ec64e5b3b80f7ec7/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60a88a61-3c25-48c8-84c5-b35bab4a058a/volumes/kubernetes.io~projected/kube-api-access-p8hp2 DeviceMajor:0 DeviceMinor:248 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a261f8f4d5b048da48aa0ba4f41bfa16d0d1540b67f421a7e6e9a95ac429c07/userdata/shm DeviceMajor:0 DeviceMinor:478 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:436 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87ba1850a1899bb73ae42d0233d441fb994378aeee19d0c4cec52bad58abf701/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:329 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9f1e9c6bb635bc9663b6fad1d374d211b92838f8bcf96afa913ecde4e5147e0/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09b6fe90-3962-49ef-bc16-486d09469b22/volumes/kubernetes.io~projected/kube-api-access-mrkbx DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:243 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:434 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fce77b3b-df3e-4d04-945c-68dc11edc61f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/83545835-dd10-4f8b-8015-25c579bdac3a/volumes/kubernetes.io~projected/kube-api-access-w5q2d DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a3430d1e2186a2de5e9637a00a4c6d50862f1804bfb8fed6b6086c0a2cf528c/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c8c1de77cf6c4cbda32164f7696a993e56def4e4db1de3840cb5ab2f7663344/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29ccd7fa19dcd4c1dfac94a30a27d98cce2a48b7f3a8782fc46acb5e8f2994a8/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31e0a09f-ff45-482d-afec-c57bd2d41cb4/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:430 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fce77b3b-df3e-4d04-945c-68dc11edc61f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:331 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2915194d-a9f7-460c-aee2-af5262da4af4/volumes/kubernetes.io~projected/kube-api-access-fv4qf DeviceMajor:0 DeviceMinor:477 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd9622a5d40c0137c96cff0882970b2c6cb0a143371074a75280c334383125fd/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:292 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:422 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9/volumes/kubernetes.io~projected/kube-api-access-x6fp9 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60a88a61-3c25-48c8-84c5-b35bab4a058a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:245 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/109981ff-df1e-4279-bc7a-1e2b09f30ad1/volumes/kubernetes.io~projected/kube-api-access-2gqh8 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b8f3882-2505-491f-8139-1905a70a5840/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b885e3c-593b-45d1-b4d4-4604ca511662/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:355 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b50ebe230b9d7b6f04e8d6ea27abcbb07763957be846c73dbf54938777b177c/userdata/shm DeviceMajor:0 DeviceMinor:360 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/050eebb4-7228-4189-b9ec-5bac3d7ed58e/volumes/kubernetes.io~projected/kube-api-access-x6ctl DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09b6fe90-3962-49ef-bc16-486d09469b22/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5b885e3c-593b-45d1-b4d4-4604ca511662/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:357 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:431 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/050eebb4-7228-4189-b9ec-5bac3d7ed58e/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:470 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01b39fe67c13da38f480a2f3596be2da5730a4eac612b80e44670dd6127c08c6/userdata/shm DeviceMajor:0 DeviceMinor:475 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~projected/kube-api-access-h2wdq DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f142aab910f0986998f5a4fdb8b0bc68f393b6aaee1b89d1cbe79f308e2195af/userdata/shm DeviceMajor:0 DeviceMinor:361 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ecf7cf31b72f03a1a62c9eb61c6584dd6751f35a49c9bf5e44d9b9433ee9f31f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91404ee3622320376e4465fbd87101539743f97d8873143673c38ae1a64368fe/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:433 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b05c2ce173dda96624c02c816fa0f7b5b5baeca0930eb0255007fd6da3fe6dd/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/741107a5-c4db-4ba9-929c-58952377929f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/153896f3b4cb172c890a65850fdd9f5159745eac48318cf3426b69f348ecc283/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31e0a09f-ff45-482d-afec-c57bd2d41cb4/volumes/kubernetes.io~projected/kube-api-access-9xdcg DeviceMajor:0 DeviceMinor:238 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:332 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0b497ad2-5bd4-4b7b-881d-d94c15fda49c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:474 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8915de45-eb8f-451b-8011-2cbf588de9b5/volumes/kubernetes.io~projected/kube-api-access-n2r9g DeviceMajor:0 DeviceMinor:333 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:01b39fe67c13da3 MacAddress:0a:a5:74:60:f1:96 Speed:10000 Mtu:8901} {Name:1a261f8f4d5b048 MacAddress:ae:4c:4d:0c:de:0a Speed:10000 Mtu:8901} {Name:2159f410d7248ba MacAddress:fe:44:bc:64:b1:d1 Speed:10000 Mtu:8901} {Name:3b50ebe230b9d7b MacAddress:02:00:94:d3:0b:13 Speed:10000 Mtu:8901} {Name:7d83332a9c55d33 MacAddress:06:5e:19:0c:d9:6a Speed:10000 Mtu:8901} {Name:7db93a08012b3ad MacAddress:56:81:87:31:32:c2 Speed:10000 Mtu:8901} {Name:9b05c2ce173dda9 MacAddress:fa:7e:4a:e9:a4:11 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:de:d3:48:7f Speed:0 Mtu:9001} {Name:br-int MacAddress:22:88:99:8d:8f:3b Speed:0 Mtu:8901} {Name:cbad41beb16322a MacAddress:a2:c9:9a:35:ae:51 Speed:10000 Mtu:8901} {Name:cd9622a5d40c013 MacAddress:32:66:00:24:4b:1c Speed:10000 Mtu:8901} {Name:d751490f72a625e MacAddress:da:8a:f1:1a:50:ba Speed:10000 Mtu:8901} {Name:e1c4b32f926d9b2 MacAddress:8a:33:cc:c5:aa:aa Speed:10000 Mtu:8901} {Name:e8176bf5fb9c579 MacAddress:aa:bf:22:2a:ab:d3 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:de:d3:48:7f Speed:0 Mtu:9001} {Name:f142aab910f0986 MacAddress:e2:9c:d7:bc:77:7a Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:a2:27:da:17:b9:fb Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ce:a5:86:c7:81:78 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 12:52:23.430135 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.429835 7010 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 12:52:23.430135 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.429903 7010 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 12:52:23.430286 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.430259 7010 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 12:52:23.430423 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.430286 7010 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-103.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 12:52:23.430464 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.430425 7010 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 12:52:23.430464 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.430434 7010 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 12:52:23.430464 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.430445 7010 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:52:23.430562 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.430467 7010 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 12:52:23.430591 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.430585 7010 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 12:52:23.430782 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.430771 7010 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 12:52:23.430907 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.430897 7010 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 12:52:23.430950 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.430934 7010 kubelet.go:493] "Attempting to sync node with API server" Oct 30 12:52:23.430950 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.430947 7010 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 12:52:23.431032 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.430956 7010 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 12:52:23.431032 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.430967 7010 kubelet.go:405] "Adding apiserver pod source" Oct 30 12:52:23.431032 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.430975 7010 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 12:52:23.431701 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.431685 7010 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 12:52:23.431701 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.431700 7010 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:52:23.432257 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.432238 7010 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 12:52:23.434951 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.434933 7010 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 12:52:23.434951 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.434947 7010 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 12:52:23.435062 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.435043 7010 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 12:52:23.435253 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.435242 7010 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 12:52:23.435289 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.435271 7010 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 12:52:23.435375 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.435362 7010 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 12:52:23.435375 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.435376 7010 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 12:52:23.435475 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.435382 7010 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 12:52:23.435475 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.435388 7010 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 12:52:23.435475 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.435394 7010 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 12:52:23.435475 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.435400 7010 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 12:52:23.435475 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.435427 7010 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 12:52:23.435475 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.435434 7010 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 12:52:23.435475 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.435440 7010 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 12:52:23.435475 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.435446 7010 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 12:52:23.435475 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.435461 7010 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 12:52:23.435475 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.435470 7010 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 12:52:23.435911 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.435521 7010 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 12:52:23.435911 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.435528 7010 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 12:52:23.438906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.438893 7010 server.go:1263] "Started kubelet" Oct 30 12:52:23.438998 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.438966 7010 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 12:52:23.439521 ip-10-0-138-103 systemd[1]: Started Kubernetes Kubelet. Oct 30 12:52:23.440329 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.440282 7010 server.go:317] "Adding debug handlers to kubelet server" Oct 30 12:52:23.440426 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.440336 7010 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 12:52:23.440480 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.440424 7010 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 12:52:23.440870 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.440847 7010 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 12:52:23.446132 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.446110 7010 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 12:52:23.448738 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.446226 7010 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 12:52:23.448738 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.446302 7010 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 12:52:23.448738 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.446319 7010 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 12:52:23.448738 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:23.446550 7010 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-103.ec2.internal\" not found" Oct 30 12:52:23.448738 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.446561 7010 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 12:52:23.448738 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.446296 7010 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:46:14 +0000 UTC" deadline="2028-08-10 13:13:06.726223778 +0000 UTC" Oct 30 12:52:23.448738 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.446593 7010 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15600h20m43.279635724s" Oct 30 12:52:23.448738 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.448599 7010 factory.go:153] Registering CRI-O factory Oct 30 12:52:23.448738 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.448618 7010 factory.go:223] Registration of the crio container factory successfully Oct 30 12:52:23.448738 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.448677 7010 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 12:52:23.448738 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.448687 7010 factory.go:55] Registering systemd factory Oct 30 12:52:23.448738 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.448695 7010 factory.go:223] Registration of the systemd container factory successfully Oct 30 12:52:23.448738 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.448713 7010 factory.go:103] Registering Raw factory Oct 30 12:52:23.448738 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.448726 7010 manager.go:1199] Started watching for new ooms in manager Oct 30 12:52:23.449560 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449148 7010 manager.go:319] Starting recovery of all containers Oct 30 12:52:23.449653 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449603 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" volumeName="kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-main-db" seLinuxMountContext="" Oct 30 12:52:23.449712 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449662 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8915de45-eb8f-451b-8011-2cbf588de9b5" volumeName="kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 12:52:23.449712 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449677 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94b7ca3d-22f2-4465-81a7-3a2d4082a6e9" volumeName="kubernetes.io/projected/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-kube-api-access-x6fp9" seLinuxMountContext="" Oct 30 12:52:23.449712 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449689 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b663748-9129-4d2c-b900-c5ddde8b4b63" volumeName="kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-cni-binary-copy" seLinuxMountContext="" Oct 30 12:52:23.449712 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449700 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1" volumeName="kubernetes.io/configmap/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-metrics-client-ca" seLinuxMountContext="" Oct 30 12:52:23.449898 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449712 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 12:52:23.449898 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449726 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fce77b3b-df3e-4d04-945c-68dc11edc61f" volumeName="kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-env-overrides" seLinuxMountContext="" Oct 30 12:52:23.449898 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449739 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fce77b3b-df3e-4d04-945c-68dc11edc61f" volumeName="kubernetes.io/secret/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 12:52:23.449898 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449754 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" volumeName="kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-out" seLinuxMountContext="" Oct 30 12:52:23.449898 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449765 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" volumeName="kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-web-config" seLinuxMountContext="" Oct 30 12:52:23.449898 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449779 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="741107a5-c4db-4ba9-929c-58952377929f" volumeName="kubernetes.io/secret/741107a5-c4db-4ba9-929c-58952377929f-agent-certs" seLinuxMountContext="" Oct 30 12:52:23.449898 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449791 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1" volumeName="kubernetes.io/secret/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 12:52:23.449898 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449802 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fce77b3b-df3e-4d04-945c-68dc11edc61f" volumeName="kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovnkube-script-lib" seLinuxMountContext="" Oct 30 12:52:23.449898 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449815 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b497ad2-5bd4-4b7b-881d-d94c15fda49c" volumeName="kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret" seLinuxMountContext="" Oct 30 12:52:23.449898 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449830 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" volumeName="kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 12:52:23.449898 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449841 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" volumeName="kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-kube-api-access-h2wdq" seLinuxMountContext="" Oct 30 12:52:23.449898 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449854 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" volumeName="kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-cluster-tls-config" seLinuxMountContext="" Oct 30 12:52:23.449898 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449865 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" volumeName="kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-volume" seLinuxMountContext="" Oct 30 12:52:23.449898 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449876 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd" volumeName="kubernetes.io/projected/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-kube-api-access-5kwws" seLinuxMountContext="" Oct 30 12:52:23.449898 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449887 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b885e3c-593b-45d1-b4d4-4604ca511662" volumeName="kubernetes.io/empty-dir/5b885e3c-593b-45d1-b4d4-4604ca511662-audit-log" seLinuxMountContext="" Oct 30 12:52:23.449898 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449898 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b885e3c-593b-45d1-b4d4-4604ca511662" volumeName="kubernetes.io/projected/5b885e3c-593b-45d1-b4d4-4604ca511662-kube-api-access-flgvz" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449909 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" volumeName="kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-metrics-client-ca" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449932 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="741107a5-c4db-4ba9-929c-58952377929f" volumeName="kubernetes.io/configmap/741107a5-c4db-4ba9-929c-58952377929f-konnectivity-ca" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449943 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8915de45-eb8f-451b-8011-2cbf588de9b5" volumeName="kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449955 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b663748-9129-4d2c-b900-c5ddde8b4b63" volumeName="kubernetes.io/projected/9b663748-9129-4d2c-b900-c5ddde8b4b63-kube-api-access-dvg8b" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.449971 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb39553b-ccf2-4cb0-af1c-339373ec1b13" volumeName="kubernetes.io/projected/bb39553b-ccf2-4cb0-af1c-339373ec1b13-kube-api-access-bdp4l" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450002 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450011 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450026 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450043 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1069a5ce-5720-4e0d-8bb8-cc874020e3dd" volumeName="kubernetes.io/empty-dir/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-tmp-dir" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450056 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e0a09f-ff45-482d-afec-c57bd2d41cb4" volumeName="kubernetes.io/projected/31e0a09f-ff45-482d-afec-c57bd2d41cb4-kube-api-access-9xdcg" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450069 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60a88a61-3c25-48c8-84c5-b35bab4a058a" volumeName="kubernetes.io/configmap/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450095 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1" volumeName="kubernetes.io/configmap/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450107 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1" volumeName="kubernetes.io/secret/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450119 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb39553b-ccf2-4cb0-af1c-339373ec1b13" volumeName="kubernetes.io/secret/bb39553b-ccf2-4cb0-af1c-339373ec1b13-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450131 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450143 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450158 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="050eebb4-7228-4189-b9ec-5bac3d7ed58e" volumeName="kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450170 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e0a09f-ff45-482d-afec-c57bd2d41cb4" volumeName="kubernetes.io/configmap/31e0a09f-ff45-482d-afec-c57bd2d41cb4-metrics-client-ca" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450183 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b8f3882-2505-491f-8139-1905a70a5840" volumeName="kubernetes.io/projected/6b8f3882-2505-491f-8139-1905a70a5840-kube-api-access-p5bqz" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450196 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b8f3882-2505-491f-8139-1905a70a5840" volumeName="kubernetes.io/secret/6b8f3882-2505-491f-8139-1905a70a5840-cert" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450210 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b663748-9129-4d2c-b900-c5ddde8b4b63" volumeName="kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450223 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-tls-assets" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450237 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-web-config" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450250 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="109981ff-df1e-4279-bc7a-1e2b09f30ad1" volumeName="kubernetes.io/empty-dir/109981ff-df1e-4279-bc7a-1e2b09f30ad1-tmp-dir" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450262 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2915194d-a9f7-460c-aee2-af5262da4af4" volumeName="kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450275 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60a88a61-3c25-48c8-84c5-b35bab4a058a" volumeName="kubernetes.io/secret/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450288 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8915de45-eb8f-451b-8011-2cbf588de9b5" volumeName="kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450301 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450314 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450327 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09b6fe90-3962-49ef-bc16-486d09469b22" volumeName="kubernetes.io/projected/09b6fe90-3962-49ef-bc16-486d09469b22-kube-api-access-mrkbx" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450341 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1069a5ce-5720-4e0d-8bb8-cc874020e3dd" volumeName="kubernetes.io/projected/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-kube-api-access-wdpc7" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450354 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="109981ff-df1e-4279-bc7a-1e2b09f30ad1" volumeName="kubernetes.io/projected/109981ff-df1e-4279-bc7a-1e2b09f30ad1-kube-api-access-2gqh8" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450368 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" volumeName="kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450381 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60a88a61-3c25-48c8-84c5-b35bab4a058a" volumeName="kubernetes.io/configmap/60a88a61-3c25-48c8-84c5-b35bab4a058a-metrics-client-ca" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450395 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8915de45-eb8f-451b-8011-2cbf588de9b5" volumeName="kubernetes.io/projected/8915de45-eb8f-451b-8011-2cbf588de9b5-kube-api-access-n2r9g" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450456 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94b7ca3d-22f2-4465-81a7-3a2d4082a6e9" volumeName="kubernetes.io/configmap/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-cni-binary-copy" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450470 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94b7ca3d-22f2-4465-81a7-3a2d4082a6e9" volumeName="kubernetes.io/configmap/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-daemon-config" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450482 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd" volumeName="kubernetes.io/empty-dir/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-tmp" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450509 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b885e3c-593b-45d1-b4d4-4604ca511662" volumeName="kubernetes.io/configmap/5b885e3c-593b-45d1-b4d4-4604ca511662-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450521 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b885e3c-593b-45d1-b4d4-4604ca511662" volumeName="kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-client-ca-bundle" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450533 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8915de45-eb8f-451b-8011-2cbf588de9b5" volumeName="kubernetes.io/configmap/8915de45-eb8f-451b-8011-2cbf588de9b5-metrics-client-ca" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450548 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450560 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60a88a61-3c25-48c8-84c5-b35bab4a058a" volumeName="kubernetes.io/empty-dir/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-textfile" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450572 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60a88a61-3c25-48c8-84c5-b35bab4a058a" volumeName="kubernetes.io/secret/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-tls" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450584 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8915de45-eb8f-451b-8011-2cbf588de9b5" volumeName="kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450595 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bc2e7c1-04ef-439a-90e5-57c4c200c62f" volumeName="kubernetes.io/projected/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-kube-api-access-4dpcr" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450607 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450620 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-kube-api-access-k9dqr" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450633 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fce77b3b-df3e-4d04-945c-68dc11edc61f" volumeName="kubernetes.io/projected/fce77b3b-df3e-4d04-945c-68dc11edc61f-kube-api-access-c62x8" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450643 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e0a09f-ff45-482d-afec-c57bd2d41cb4" volumeName="kubernetes.io/secret/31e0a09f-ff45-482d-afec-c57bd2d41cb4-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450651 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83545835-dd10-4f8b-8015-25c579bdac3a" volumeName="kubernetes.io/projected/83545835-dd10-4f8b-8015-25c579bdac3a-kube-api-access-w5q2d" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450662 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8915de45-eb8f-451b-8011-2cbf588de9b5" volumeName="kubernetes.io/empty-dir/8915de45-eb8f-451b-8011-2cbf588de9b5-tmp" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450672 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" volumeName="kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450682 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60a88a61-3c25-48c8-84c5-b35bab4a058a" volumeName="kubernetes.io/projected/60a88a61-3c25-48c8-84c5-b35bab4a058a-kube-api-access-p8hp2" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450694 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1" volumeName="kubernetes.io/empty-dir/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-volume-directive-shadow" seLinuxMountContext="" Oct 30 12:52:23.450663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450705 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-grpc-tls" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450717 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="050eebb4-7228-4189-b9ec-5bac3d7ed58e" volumeName="kubernetes.io/projected/050eebb4-7228-4189-b9ec-5bac3d7ed58e-kube-api-access-x6ctl" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450735 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1069a5ce-5720-4e0d-8bb8-cc874020e3dd" volumeName="kubernetes.io/configmap/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-config-volume" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450744 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1069a5ce-5720-4e0d-8bb8-cc874020e3dd" volumeName="kubernetes.io/secret/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-metrics-tls" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450751 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" volumeName="kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450760 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd" volumeName="kubernetes.io/empty-dir/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-tuned" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450767 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e0a09f-ff45-482d-afec-c57bd2d41cb4" volumeName="kubernetes.io/secret/31e0a09f-ff45-482d-afec-c57bd2d41cb4-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450774 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="404bb84d-f044-488e-a221-389f71e61181" volumeName="kubernetes.io/projected/404bb84d-f044-488e-a221-389f71e61181-kube-api-access-hd6nk" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450785 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b885e3c-593b-45d1-b4d4-4604ca511662" volumeName="kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450797 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" volumeName="kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-tls-assets" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450804 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" volumeName="kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450811 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b885e3c-593b-45d1-b4d4-4604ca511662" volumeName="kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450818 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83545835-dd10-4f8b-8015-25c579bdac3a" volumeName="kubernetes.io/configmap/83545835-dd10-4f8b-8015-25c579bdac3a-iptables-alerter-script" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450825 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8915de45-eb8f-451b-8011-2cbf588de9b5" volumeName="kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-grpc-tls" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450833 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450843 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-config-out" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450853 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-config" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450860 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8915de45-eb8f-451b-8011-2cbf588de9b5" volumeName="kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450869 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b663748-9129-4d2c-b900-c5ddde8b4b63" volumeName="kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450877 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1" volumeName="kubernetes.io/projected/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-api-access-7llpr" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450884 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-db" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450892 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450901 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09b6fe90-3962-49ef-bc16-486d09469b22" volumeName="kubernetes.io/configmap/09b6fe90-3962-49ef-bc16-486d09469b22-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450913 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09b6fe90-3962-49ef-bc16-486d09469b22" volumeName="kubernetes.io/empty-dir/09b6fe90-3962-49ef-bc16-486d09469b22-data-volume" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450925 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09b6fe90-3962-49ef-bc16-486d09469b22" volumeName="kubernetes.io/secret/09b6fe90-3962-49ef-bc16-486d09469b22-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450937 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b885e3c-593b-45d1-b4d4-4604ca511662" volumeName="kubernetes.io/configmap/5b885e3c-593b-45d1-b4d4-4604ca511662-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450948 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bc2e7c1-04ef-439a-90e5-57c4c200c62f" volumeName="kubernetes.io/configmap/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-serviceca" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450960 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd51364d-3838-4a3d-b470-ccf6483121e7" volumeName="kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450972 7010 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fce77b3b-df3e-4d04-945c-68dc11edc61f" volumeName="kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovnkube-config" seLinuxMountContext="" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450984 7010 reconstruct.go:97] "Volume reconstruction finished" Oct 30 12:52:23.453377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.450991 7010 reconciler.go:29] "Reconciler: start to sync state" Oct 30 12:52:23.464116 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.463918 7010 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 12:52:23.480432 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.480401 7010 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 12:52:23.480432 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.480426 7010 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 12:52:23.480432 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.480442 7010 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 12:52:23.480743 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:23.480475 7010 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 12:52:23.483351 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.483322 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 12:52:23.484755 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.484223 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 12:52:23.492504 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.492458 7010 generic.go:356] "Generic (PLEG): container finished" podID="60a88a61-3c25-48c8-84c5-b35bab4a058a" containerID="9ae6070703f12858dc766936e59bec535b53fa4de51d52e074ce3edf46b7bd82" exitCode=0 Oct 30 12:52:23.497250 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.497229 7010 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:52:23.497364 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.497351 7010 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 12:52:23.499374 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.499354 7010 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:52:23.522119 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.522091 7010 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:52:23.522729 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.522703 7010 generic.go:356] "Generic (PLEG): container finished" podID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerID="9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce" exitCode=0 Oct 30 12:52:23.523060 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.523040 7010 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 12:52:23.526652 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.526626 7010 generic.go:356] "Generic (PLEG): container finished" podID="9b663748-9129-4d2c-b900-c5ddde8b4b63" containerID="fd53f8b4fc42c0bf908590bdf9625ccd5b14913b94e5ec9ce864f8e67bd75180" exitCode=0 Oct 30 12:52:23.526652 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.526652 7010 generic.go:356] "Generic (PLEG): container finished" podID="9b663748-9129-4d2c-b900-c5ddde8b4b63" containerID="aadc12dcbf14159d5c5971a7384dd5a9231d3fd7ee273d694e75559036c56382" exitCode=0 Oct 30 12:52:23.526782 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.526661 7010 generic.go:356] "Generic (PLEG): container finished" podID="9b663748-9129-4d2c-b900-c5ddde8b4b63" containerID="4028df408a3169f2ee57728e8c745e0dc26415cadade192eea29acf91bf20202" exitCode=0 Oct 30 12:52:23.526782 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.526669 7010 generic.go:356] "Generic (PLEG): container finished" podID="9b663748-9129-4d2c-b900-c5ddde8b4b63" containerID="3f255ca524212386aeb4ced3023c846626c9153b11f426df68d387624cefbbb2" exitCode=0 Oct 30 12:52:23.526782 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.526677 7010 generic.go:356] "Generic (PLEG): container finished" podID="9b663748-9129-4d2c-b900-c5ddde8b4b63" containerID="88130d417b5279da3d0120aae1115f03a4011e6f463a42297a6862de0ba7307b" exitCode=0 Oct 30 12:52:23.526782 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.526686 7010 generic.go:356] "Generic (PLEG): container finished" podID="9b663748-9129-4d2c-b900-c5ddde8b4b63" containerID="883447b0e26d5c770faead762bc31a984bb0eb4243330d4631616c13c3050bb8" exitCode=0 Oct 30 12:52:23.530471 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.530454 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 12:52:23.530871 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.530845 7010 generic.go:356] "Generic (PLEG): container finished" podID="fce77b3b-df3e-4d04-945c-68dc11edc61f" containerID="9a37a3ab36d0e68cb97c45df211a97c7ae013550f314c53ef1c76b27075b7437" exitCode=1 Oct 30 12:52:23.542996 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.542966 7010 generic.go:356] "Generic (PLEG): container finished" podID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerID="9c17b806405334125a9c015df1ccb6aaf991fcea9c980a47541152a5fd6c8908" exitCode=0 Oct 30 12:52:23.546879 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.546855 7010 generic.go:356] "Generic (PLEG): container finished" podID="b4c73d6ca462e65d2a8ead7238da805c" containerID="e30a112d7a41fdcd2f677bbb475da5cd078e5a53e71ebdc49e797038e2462858" exitCode=0 Oct 30 12:52:23.573656 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.573589 7010 manager.go:324] Recovery completed Oct 30 12:52:23.580577 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:23.580554 7010 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 12:52:23.597928 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.597901 7010 cpu_manager.go:226] "Starting" policy="none" Oct 30 12:52:23.597928 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.597921 7010 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 12:52:23.598064 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.597945 7010 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 12:52:23.598124 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.598108 7010 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 12:52:23.598157 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.598127 7010 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 12:52:23.598157 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.598151 7010 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 12:52:23.598269 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.598159 7010 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 12:52:23.598269 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.598167 7010 policy_none.go:50] "Start" Oct 30 12:52:23.598269 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.598180 7010 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 12:52:23.598269 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.598193 7010 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 12:52:23.599884 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.599870 7010 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 12:52:23.599920 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.599885 7010 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 12:52:23.599920 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.599892 7010 policy_none.go:44] "Start" Oct 30 12:52:23.604580 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.604550 7010 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 12:52:23.609113 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.609098 7010 manager.go:341] "Starting Device Plugin manager" Oct 30 12:52:23.609604 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:23.609129 7010 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 12:52:23.609679 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.609608 7010 server.go:87] "Starting device plugin registration server" Oct 30 12:52:23.609931 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.609917 7010 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 12:52:23.610008 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.609931 7010 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 12:52:23.610355 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.610025 7010 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 12:52:23.610497 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.610448 7010 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 12:52:23.610558 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.610507 7010 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 12:52:23.610558 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.610516 7010 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 12:52:23.610734 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.610621 7010 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:52:23.610482774Z","UUID":"adbfbd32-e035-45fe-ad85-d4ccc95f2590","Handler":null,"Name":"","Endpoint":""} Oct 30 12:52:23.612741 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:23.612718 7010 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 12:52:23.612847 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.612747 7010 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 12:52:23.612847 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.612789 7010 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 12:52:23.710628 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.710590 7010 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 12:52:23.712801 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.712782 7010 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientMemory" Oct 30 12:52:23.712912 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.712813 7010 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 12:52:23.712912 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.712824 7010 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-103.ec2.internal" event="NodeHasSufficientPID" Oct 30 12:52:23.712912 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.712867 7010 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-103.ec2.internal" Oct 30 12:52:23.754183 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.754152 7010 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-103.ec2.internal" Oct 30 12:52:23.754284 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.754241 7010 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-103.ec2.internal" Oct 30 12:52:23.780801 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.780754 7010 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal"] Oct 30 12:52:23.781547 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.781524 7010 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:52:23.781665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.781626 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" event={"ID":"4db7be0b2dbb4b62fd854fb3bbef0e8f","Type":"ContainerStarted","Data":"7800f79398bd4aebfc552339785fe1a0fe724fd489d7b3a7c656b04ea57dbeec"} Oct 30 12:52:23.781725 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.781671 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" event={"ID":"4db7be0b2dbb4b62fd854fb3bbef0e8f","Type":"ContainerStarted","Data":"87ba1850a1899bb73ae42d0233d441fb994378aeee19d0c4cec52bad58abf701"} Oct 30 12:52:23.781725 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.781702 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" event={"ID":"b4c73d6ca462e65d2a8ead7238da805c","Type":"ContainerStarted","Data":"268c99143ecbb1f541c174b80afc75a748af78999a9789cc006abe73c7d3d1c9"} Oct 30 12:52:23.781725 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.781710 7010 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" Oct 30 12:52:23.781725 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.781722 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" event={"ID":"b4c73d6ca462e65d2a8ead7238da805c","Type":"ContainerDied","Data":"e30a112d7a41fdcd2f677bbb475da5cd078e5a53e71ebdc49e797038e2462858"} Oct 30 12:52:23.781842 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.781735 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" event={"ID":"b4c73d6ca462e65d2a8ead7238da805c","Type":"ContainerStarted","Data":"98c3ddec071d956e18ff654cccc97b8264694b1c702c2c161017e23ea91bfd27"} Oct 30 12:52:23.812525 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.812476 7010 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:52:23.812664 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:23.812550 7010 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-103.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" Oct 30 12:52:23.812664 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.812611 7010 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:52:23.812664 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:23.812647 7010 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:52:23.898962 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.898927 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4c73d6ca462e65d2a8ead7238da805c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal\" (UID: \"b4c73d6ca462e65d2a8ead7238da805c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:52:23.898962 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.898963 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4db7be0b2dbb4b62fd854fb3bbef0e8f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-103.ec2.internal\" (UID: \"4db7be0b2dbb4b62fd854fb3bbef0e8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" Oct 30 12:52:23.899154 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.898987 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4c73d6ca462e65d2a8ead7238da805c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal\" (UID: \"b4c73d6ca462e65d2a8ead7238da805c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:52:23.999625 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.999566 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4c73d6ca462e65d2a8ead7238da805c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal\" (UID: \"b4c73d6ca462e65d2a8ead7238da805c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:52:23.999754 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.999636 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4c73d6ca462e65d2a8ead7238da805c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal\" (UID: \"b4c73d6ca462e65d2a8ead7238da805c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:52:23.999754 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.999656 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4db7be0b2dbb4b62fd854fb3bbef0e8f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-103.ec2.internal\" (UID: \"4db7be0b2dbb4b62fd854fb3bbef0e8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" Oct 30 12:52:23.999754 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.999680 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4c73d6ca462e65d2a8ead7238da805c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal\" (UID: \"b4c73d6ca462e65d2a8ead7238da805c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:52:23.999754 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.999712 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4db7be0b2dbb4b62fd854fb3bbef0e8f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-103.ec2.internal\" (UID: \"4db7be0b2dbb4b62fd854fb3bbef0e8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" Oct 30 12:52:23.999909 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:23.999751 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4c73d6ca462e65d2a8ead7238da805c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal\" (UID: \"b4c73d6ca462e65d2a8ead7238da805c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:52:24.432118 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.432084 7010 apiserver.go:52] "Watching apiserver" Oct 30 12:52:24.486678 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.486645 7010 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 12:52:24.487319 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.487289 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal","openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf","openshift-monitoring/node-exporter-vdft6","openshift-cluster-node-tuning-operator/tuned-bsmf4","openshift-multus/network-metrics-daemon-b68v5","openshift-network-operator/iptables-alerter-r9nqr","kube-system/global-pull-secret-syncer-2xg75","openshift-multus/multus-additional-cni-plugins-4mmzs","openshift-dns/node-resolver-qldmf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k","kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal","openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl","openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg","openshift-ovn-kubernetes/ovnkube-node-jz7rd","openshift-dns/dns-default-mtsz8","openshift-monitoring/alertmanager-main-0","openshift-multus/multus-4tbwx","openshift-network-diagnostics/network-check-target-dgnrj","openshift-image-registry/node-ca-bm8bc","openshift-insights/insights-runtime-extractor-4hts2","kube-system/konnectivity-agent-zchsd","openshift-ingress-canary/ingress-canary-4rfmk","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/metrics-server-56b76dcb6d-fm86z","openshift-monitoring/thanos-querier-79ff6956fb-ljgd2"] Oct 30 12:52:24.509212 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.509181 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-588rd\"" Oct 30 12:52:24.509377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.509217 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-78z4l\"" Oct 30 12:52:24.509377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.509230 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d8ng4\"" Oct 30 12:52:24.509377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.509294 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 12:52:24.509377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.509342 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 12:52:24.509591 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.509394 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pdl9w\"" Oct 30 12:52:24.509591 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.509522 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n54s8\"" Oct 30 12:52:24.510409 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.510394 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.510875 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.510860 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.510943 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.510926 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.511013 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.510948 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 12:52:24.511013 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.510963 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.511013 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.510952 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.511013 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.510976 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.511013 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.510968 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 12:52:24.511013 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.510990 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.511013 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.511002 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 12:52:24.511236 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.511065 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.511236 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.511085 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 12:52:24.511236 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.511092 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 12:52:24.511236 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.511122 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 12:52:24.511236 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.511122 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 12:52:24.511372 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.511243 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 12:52:24.517816 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.517802 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.518404 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.518388 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 12:52:24.518462 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.518420 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g6csn\"" Oct 30 12:52:24.518531 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.518483 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.518657 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.518645 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.518795 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.518768 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6lqrk\"" Oct 30 12:52:24.519589 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.519571 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.519797 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.519783 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 12:52:24.519908 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.519895 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.519962 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.519900 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 12:52:24.519962 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.519939 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rnjq7\"" Oct 30 12:52:24.527616 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.527596 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 12:52:24.527720 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.527696 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.527720 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.527700 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 12:52:24.527806 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.527785 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.527842 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.527803 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 12:52:24.527842 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.527810 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.527906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.527858 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.527906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.527874 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.527979 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.527966 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.528085 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.528074 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 12:52:24.528183 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.528168 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.528225 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.528198 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 12:52:24.528271 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.528252 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 12:52:24.528343 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.528327 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 12:52:24.528343 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.528338 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 12:52:24.528415 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.528344 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 12:52:24.528415 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.528381 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 12:52:24.528522 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.528509 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 12:52:24.528562 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.528526 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 12:52:24.528616 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.528561 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 12:52:24.528616 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.528571 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 12:52:24.528848 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.528831 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fv4k6\"" Oct 30 12:52:24.540918 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.540900 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 12:52:24.541012 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.540947 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 12:52:24.541012 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.540972 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 12:52:24.541120 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541018 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 12:52:24.541120 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541048 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 12:52:24.541120 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541063 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 12:52:24.541280 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541254 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 12:52:24.541389 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541353 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 12:52:24.541389 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541382 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 12:52:24.541560 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541416 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 12:52:24.541560 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541432 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 12:52:24.541560 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541441 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 12:52:24.541560 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541521 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 12:52:24.541560 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541553 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 12:52:24.541809 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541552 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 12:52:24.541809 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541585 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 12:52:24.541809 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541610 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hfgh7\"" Oct 30 12:52:24.541809 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541645 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9fz8p\"" Oct 30 12:52:24.541809 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541673 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-22g27\"" Oct 30 12:52:24.541809 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541674 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 12:52:24.541809 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541674 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6vr75\"" Oct 30 12:52:24.541809 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541714 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 12:52:24.541809 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541715 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 12:52:24.541809 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541726 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-am8n6hsri9i07\"" Oct 30 12:52:24.541809 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541724 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hxn87\"" Oct 30 12:52:24.541809 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541738 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 12:52:24.541809 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541763 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 12:52:24.541809 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541779 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-sn4nb\"" Oct 30 12:52:24.541809 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541783 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 12:52:24.541809 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541816 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 12:52:24.542389 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541812 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 12:52:24.542389 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541855 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 12:52:24.542389 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541870 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 12:52:24.542389 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541873 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 12:52:24.542389 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541911 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-60o5qmh2r8rg1\"" Oct 30 12:52:24.542389 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.541979 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 12:52:24.542389 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542060 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 12:52:24.542389 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542092 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 12:52:24.542389 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542242 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 12:52:24.542389 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542252 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ee888a1mr1r2m\"" Oct 30 12:52:24.542389 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542314 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-l6ww5\"" Oct 30 12:52:24.542848 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542408 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 12:52:24.542848 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542472 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 12:52:24.542848 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542521 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sb6s2\"" Oct 30 12:52:24.542848 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542540 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nklv7\"" Oct 30 12:52:24.542848 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542579 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-wqldm\"" Oct 30 12:52:24.542848 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542625 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 12:52:24.542848 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542657 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kq2jq\"" Oct 30 12:52:24.542848 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542708 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hhkwq\"" Oct 30 12:52:24.542848 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542736 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 12:52:24.542848 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542758 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 12:52:24.542848 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542798 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-27xf9\"" Oct 30 12:52:24.542848 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542811 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 12:52:24.542848 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.542852 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-767qk\"" Oct 30 12:52:24.551697 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.551675 7010 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 12:52:24.554868 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.554851 7010 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:52:24.555024 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.555009 7010 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" Oct 30 12:52:24.579153 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.579131 7010 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:52:24.579237 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:24.579176 7010 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-103.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-103.ec2.internal" Oct 30 12:52:24.579928 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.579917 7010 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 12:52:24.579967 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:24.579946 7010 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-103.ec2.internal" Oct 30 12:52:24.602326 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602298 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:52:24.602416 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602333 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-textfile\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.602416 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602360 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.602416 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602387 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.602577 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602412 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.602577 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602462 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-textfile\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.602577 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602464 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-node-log\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.602721 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602619 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-sys\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.602721 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602648 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31e0a09f-ff45-482d-afec-c57bd2d41cb4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:24.602721 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602650 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.602721 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602672 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-sys-fs\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.602721 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602680 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-original-pull-secret\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:52:24.602721 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602681 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.602721 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602706 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4qf\" (UniqueName: \"kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf\") pod \"network-check-target-dgnrj\" (UID: \"2915194d-a9f7-460c-aee2-af5262da4af4\") " pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:52:24.603007 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602737 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60a88a61-3c25-48c8-84c5-b35bab4a058a-metrics-client-ca\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.603007 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602762 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.603007 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602789 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-run-netns\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.603007 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602814 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ctl\" (UniqueName: \"kubernetes.io/projected/050eebb4-7228-4189-b9ec-5bac3d7ed58e-kube-api-access-x6ctl\") pod \"network-metrics-daemon-b68v5\" (UID: \"050eebb4-7228-4189-b9ec-5bac3d7ed58e\") " pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:52:24.603007 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602833 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31e0a09f-ff45-482d-afec-c57bd2d41cb4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:24.603007 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602839 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-netns\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.603007 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602888 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-daemon-config\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.603007 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.602952 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b8f3882-2505-491f-8139-1905a70a5840-cert\") pod \"ingress-canary-4rfmk\" (UID: \"6b8f3882-2505-491f-8139-1905a70a5840\") " pod="openshift-ingress-canary/ingress-canary-4rfmk" Oct 30 12:52:24.603362 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603010 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.603362 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603008 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-serviceca\") pod \"node-ca-bm8bc\" (UID: \"8bc2e7c1-04ef-439a-90e5-57c4c200c62f\") " pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:52:24.603362 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603056 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-secret-metrics-server-client-certs\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:24.603362 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603101 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-device-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.603362 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603101 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60a88a61-3c25-48c8-84c5-b35bab4a058a-metrics-client-ca\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.603362 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603093 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-daemon-config\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.603362 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603199 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b8f3882-2505-491f-8139-1905a70a5840-cert\") pod \"ingress-canary-4rfmk\" (UID: \"6b8f3882-2505-491f-8139-1905a70a5840\") " pod="openshift-ingress-canary/ingress-canary-4rfmk" Oct 30 12:52:24.603362 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603210 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fp9\" (UniqueName: \"kubernetes.io/projected/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-kube-api-access-x6fp9\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.603362 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603247 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.603362 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603265 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-secret-metrics-server-client-certs\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:24.603362 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603269 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-host\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.603362 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603303 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-dbus\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:52:24.603362 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603357 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8hp2\" (UniqueName: \"kubernetes.io/projected/60a88a61-3c25-48c8-84c5-b35bab4a058a-kube-api-access-p8hp2\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.603923 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603389 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-system-cni-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.603923 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603416 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5bqz\" (UniqueName: \"kubernetes.io/projected/6b8f3882-2505-491f-8139-1905a70a5840-kube-api-access-p5bqz\") pod \"ingress-canary-4rfmk\" (UID: \"6b8f3882-2505-491f-8139-1905a70a5840\") " pod="openshift-ingress-canary/ingress-canary-4rfmk" Oct 30 12:52:24.603923 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603462 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.603923 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603481 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kwws\" (UniqueName: \"kubernetes.io/projected/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-kube-api-access-5kwws\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.603923 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603565 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83545835-dd10-4f8b-8015-25c579bdac3a-host-slash\") pod \"iptables-alerter-r9nqr\" (UID: \"83545835-dd10-4f8b-8015-25c579bdac3a\") " pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:52:24.603923 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603569 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.603923 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603672 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd6nk\" (UniqueName: \"kubernetes.io/projected/404bb84d-f044-488e-a221-389f71e61181-kube-api-access-hd6nk\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.603923 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603703 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-cni-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.603923 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603721 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.603923 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603782 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-serviceca\") pod \"node-ca-bm8bc\" (UID: \"8bc2e7c1-04ef-439a-90e5-57c4c200c62f\") " pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:52:24.603923 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603768 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.603923 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603831 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-slash\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.603923 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603860 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-run-ovn-kubernetes\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.603923 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603895 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.603923 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603929 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603947 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.603998 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604028 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs\") pod \"network-metrics-daemon-b68v5\" (UID: \"050eebb4-7228-4189-b9ec-5bac3d7ed58e\") " pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604029 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604072 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-wtmp\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604090 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604145 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604176 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-var-lib-kubelet\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604199 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-cnibin\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604227 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8915de45-eb8f-451b-8011-2cbf588de9b5-metrics-client-ca\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604229 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604334 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604364 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604386 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604421 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604430 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604439 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovnkube-config\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604442 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8915de45-eb8f-451b-8011-2cbf588de9b5-metrics-client-ca\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604463 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-env-overrides\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.604559 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604527 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/050eebb4-7228-4189-b9ec-5bac3d7ed58e-metrics-certs\") pod \"network-metrics-daemon-b68v5\" (UID: \"050eebb4-7228-4189-b9ec-5bac3d7ed58e\") " pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604638 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-env-overrides\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604637 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604671 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovnkube-config\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604691 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604729 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31e0a09f-ff45-482d-afec-c57bd2d41cb4-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604765 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604790 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604822 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-log-socket\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604850 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dpcr\" (UniqueName: \"kubernetes.io/projected/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-kube-api-access-4dpcr\") pod \"node-ca-bm8bc\" (UID: \"8bc2e7c1-04ef-439a-90e5-57c4c200c62f\") " pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604910 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604913 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-metrics-tls\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604950 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604962 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604988 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31e0a09f-ff45-482d-afec-c57bd2d41cb4-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.604956 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31e0a09f-ff45-482d-afec-c57bd2d41cb4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605030 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-multus-certs\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605047 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605094 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-etc-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605112 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysctl-d\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605129 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09b6fe90-3962-49ef-bc16-486d09469b22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605134 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-metrics-tls\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605146 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdpc7\" (UniqueName: \"kubernetes.io/projected/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-kube-api-access-wdpc7\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605178 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-cni-binary-copy\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605195 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31e0a09f-ff45-482d-afec-c57bd2d41cb4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605225 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2r9g\" (UniqueName: \"kubernetes.io/projected/8915de45-eb8f-451b-8011-2cbf588de9b5-kube-api-access-n2r9g\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605258 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605284 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605401 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-cni-binary-copy\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605430 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605445 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605479 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2wdq\" (UniqueName: \"kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-kube-api-access-h2wdq\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605517 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-kubernetes\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605517 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09b6fe90-3962-49ef-bc16-486d09469b22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:24.605507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605534 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-socket-dir-parent\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605576 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-cni-multus\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605614 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-etc-kubernetes\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605642 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-systemd-units\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605668 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/109981ff-df1e-4279-bc7a-1e2b09f30ad1-hosts-file\") pod \"node-resolver-qldmf\" (UID: \"109981ff-df1e-4279-bc7a-1e2b09f30ad1\") " pod="openshift-dns/node-resolver-qldmf" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605707 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-client-ca-bundle\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605739 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605758 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605856 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-conf-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605935 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605950 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-client-ca-bundle\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605974 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-config\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.605999 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606031 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c62x8\" (UniqueName: \"kubernetes.io/projected/fce77b3b-df3e-4d04-945c-68dc11edc61f-kube-api-access-c62x8\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606070 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-etc-selinux\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606091 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xdcg\" (UniqueName: \"kubernetes.io/projected/31e0a09f-ff45-482d-afec-c57bd2d41cb4-kube-api-access-9xdcg\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606121 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606190 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-tls\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606292 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-volume\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606346 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-web-config\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606472 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-tls\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606476 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-config\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606533 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-volume\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606642 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-web-config\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606663 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-tls\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606702 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-cni-bin\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606731 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606759 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-k8s-cni-cncf-io\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606784 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606814 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-thanos-querier-tls\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606830 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-var-lib-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606853 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-systemd\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.606866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606877 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-lib-modules\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606899 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-sys\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606924 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/109981ff-df1e-4279-bc7a-1e2b09f30ad1-tmp-dir\") pod \"node-resolver-qldmf\" (UID: \"109981ff-df1e-4279-bc7a-1e2b09f30ad1\") " pod="openshift-dns/node-resolver-qldmf" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606949 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gqh8\" (UniqueName: \"kubernetes.io/projected/109981ff-df1e-4279-bc7a-1e2b09f30ad1-kube-api-access-2gqh8\") pod \"node-resolver-qldmf\" (UID: \"109981ff-df1e-4279-bc7a-1e2b09f30ad1\") " pod="openshift-dns/node-resolver-qldmf" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.606976 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b885e3c-593b-45d1-b4d4-4604ca511662-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607001 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-tmp-dir\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607031 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607055 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-cni-binary-copy\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607080 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-socket-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607097 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607106 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607142 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovn-node-metrics-cert\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607160 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysconfig\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607177 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysctl-conf\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607192 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-tuned\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607208 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607229 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607252 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-grpc-tls\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607285 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607305 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-kubelet-config\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607333 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7llpr\" (UniqueName: \"kubernetes.io/projected/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-api-access-7llpr\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607354 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607363 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607377 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-cni-bin\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607394 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9dqr\" (UniqueName: \"kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-kube-api-access-k9dqr\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607430 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-ovn\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607729 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607755 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607799 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-tuned\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607879 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.607943 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608036 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovn-node-metrics-cert\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608087 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b885e3c-593b-45d1-b4d4-4604ca511662-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608131 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/109981ff-df1e-4279-bc7a-1e2b09f30ad1-tmp-dir\") pod \"node-resolver-qldmf\" (UID: \"109981ff-df1e-4279-bc7a-1e2b09f30ad1\") " pod="openshift-dns/node-resolver-qldmf" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608149 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608175 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-cni-binary-copy\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608246 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-tmp-dir\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608255 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608285 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5q2d\" (UniqueName: \"kubernetes.io/projected/83545835-dd10-4f8b-8015-25c579bdac3a-kube-api-access-w5q2d\") pod \"iptables-alerter-r9nqr\" (UID: \"83545835-dd10-4f8b-8015-25c579bdac3a\") " pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608301 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/741107a5-c4db-4ba9-929c-58952377929f-agent-certs\") pod \"konnectivity-agent-zchsd\" (UID: \"741107a5-c4db-4ba9-929c-58952377929f\") " pod="kube-system/konnectivity-agent-zchsd" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608335 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5b885e3c-593b-45d1-b4d4-4604ca511662-audit-log\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608352 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-root\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608368 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608383 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.608395 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608413 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608509 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8915de45-eb8f-451b-8011-2cbf588de9b5-secret-grpc-tls\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608520 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608544 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-secret-metrics-server-tls\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608562 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-kubelet\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608569 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5b885e3c-593b-45d1-b4d4-4604ca511662-audit-log\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608580 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608599 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/741107a5-c4db-4ba9-929c-58952377929f-agent-certs\") pod \"konnectivity-agent-zchsd\" (UID: \"741107a5-c4db-4ba9-929c-58952377929f\") " pod="kube-system/konnectivity-agent-zchsd" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608712 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-out\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608805 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608821 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608864 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5b885e3c-593b-45d1-b4d4-4604ca511662-secret-metrics-server-tls\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608888 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608872 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-out\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608903 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-systemd\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608933 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-cni-netd\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608962 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83545835-dd10-4f8b-8015-25c579bdac3a-iptables-alerter-script\") pod \"iptables-alerter-r9nqr\" (UID: \"83545835-dd10-4f8b-8015-25c579bdac3a\") " pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.608973 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b663748-9129-4d2c-b900-c5ddde8b4b63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609002 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flgvz\" (UniqueName: \"kubernetes.io/projected/5b885e3c-593b-45d1-b4d4-4604ca511662-kube-api-access-flgvz\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609080 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83545835-dd10-4f8b-8015-25c579bdac3a-iptables-alerter-script\") pod \"iptables-alerter-r9nqr\" (UID: \"83545835-dd10-4f8b-8015-25c579bdac3a\") " pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609113 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvg8b\" (UniqueName: \"kubernetes.io/projected/9b663748-9129-4d2c-b900-c5ddde8b4b63-kube-api-access-dvg8b\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609139 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609158 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609199 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-registration-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609269 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-config-volume\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609290 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-modprobe-d\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609316 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09b6fe90-3962-49ef-bc16-486d09469b22-crio-socket\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609332 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609340 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609392 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609462 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609456 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-config-volume\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609460 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609541 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb39553b-ccf2-4cb0-af1c-339373ec1b13-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-g2zfl\" (UID: \"bb39553b-ccf2-4cb0-af1c-339373ec1b13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609594 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09b6fe90-3962-49ef-bc16-486d09469b22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609624 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovnkube-script-lib\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609640 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/741107a5-c4db-4ba9-929c-58952377929f-konnectivity-ca\") pod \"konnectivity-agent-zchsd\" (UID: \"741107a5-c4db-4ba9-929c-58952377929f\") " pod="kube-system/konnectivity-agent-zchsd" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609682 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-kubelet\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609706 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bb39553b-ccf2-4cb0-af1c-339373ec1b13-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-g2zfl\" (UID: \"bb39553b-ccf2-4cb0-af1c-339373ec1b13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609721 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609779 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-system-cni-dir\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609813 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-tmp\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609831 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09b6fe90-3962-49ef-bc16-486d09469b22-data-volume\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609837 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fce77b3b-df3e-4d04-945c-68dc11edc61f-ovnkube-script-lib\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609849 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrkbx\" (UniqueName: \"kubernetes.io/projected/09b6fe90-3962-49ef-bc16-486d09469b22-kube-api-access-mrkbx\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609864 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09b6fe90-3962-49ef-bc16-486d09469b22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609850 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/741107a5-c4db-4ba9-929c-58952377929f-konnectivity-ca\") pod \"konnectivity-agent-zchsd\" (UID: \"741107a5-c4db-4ba9-929c-58952377929f\") " pod="kube-system/konnectivity-agent-zchsd" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609877 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609893 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-tmp\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609904 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609910 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09b6fe90-3962-49ef-bc16-486d09469b22-data-volume\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609930 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-hostroot\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609954 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609971 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-cnibin\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.609987 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-run\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.609993 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.610009 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-host\") pod \"node-ca-bm8bc\" (UID: \"8bc2e7c1-04ef-439a-90e5-57c4c200c62f\") " pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:52:24.611658 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.610034 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5b885e3c-593b-45d1-b4d4-4604ca511662-metrics-server-audit-profiles\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:24.611658 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.610067 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-kubelet-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.611658 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.610069 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.611658 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.610095 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.611658 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.610113 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdp4l\" (UniqueName: \"kubernetes.io/projected/bb39553b-ccf2-4cb0-af1c-339373ec1b13-kube-api-access-bdp4l\") pod \"monitoring-plugin-77f4f76594-g2zfl\" (UID: \"bb39553b-ccf2-4cb0-af1c-339373ec1b13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" Oct 30 12:52:24.611658 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.610160 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.611658 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.610222 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.611658 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.610267 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5b885e3c-593b-45d1-b4d4-4604ca511662-metrics-server-audit-profiles\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:24.611658 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.610269 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8915de45-eb8f-451b-8011-2cbf588de9b5-tmp\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.611658 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.610354 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8915de45-eb8f-451b-8011-2cbf588de9b5-tmp\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.611658 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.610406 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.626983 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.626966 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv4qf\" (UniqueName: \"kubernetes.io/projected/2915194d-a9f7-460c-aee2-af5262da4af4-kube-api-access-fv4qf\") pod \"network-check-target-dgnrj\" (UID: \"2915194d-a9f7-460c-aee2-af5262da4af4\") " pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:52:24.628029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.628012 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6ctl\" (UniqueName: \"kubernetes.io/projected/050eebb4-7228-4189-b9ec-5bac3d7ed58e-kube-api-access-x6ctl\") pod \"network-metrics-daemon-b68v5\" (UID: \"050eebb4-7228-4189-b9ec-5bac3d7ed58e\") " pod="openshift-multus/network-metrics-daemon-b68v5" Oct 30 12:52:24.628381 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.628365 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8hp2\" (UniqueName: \"kubernetes.io/projected/60a88a61-3c25-48c8-84c5-b35bab4a058a-kube-api-access-p8hp2\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.628927 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.628913 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5bqz\" (UniqueName: \"kubernetes.io/projected/6b8f3882-2505-491f-8139-1905a70a5840-kube-api-access-p5bqz\") pod \"ingress-canary-4rfmk\" (UID: \"6b8f3882-2505-491f-8139-1905a70a5840\") " pod="openshift-ingress-canary/ingress-canary-4rfmk" Oct 30 12:52:24.629708 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.629689 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6fp9\" (UniqueName: \"kubernetes.io/projected/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-kube-api-access-x6fp9\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.631416 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.631401 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kwws\" (UniqueName: \"kubernetes.io/projected/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-kube-api-access-5kwws\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.637481 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.637460 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd6nk\" (UniqueName: \"kubernetes.io/projected/404bb84d-f044-488e-a221-389f71e61181-kube-api-access-hd6nk\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.640703 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.640683 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dpcr\" (UniqueName: \"kubernetes.io/projected/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-kube-api-access-4dpcr\") pod \"node-ca-bm8bc\" (UID: \"8bc2e7c1-04ef-439a-90e5-57c4c200c62f\") " pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:52:24.641069 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.641042 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdpc7\" (UniqueName: \"kubernetes.io/projected/1069a5ce-5720-4e0d-8bb8-cc874020e3dd-kube-api-access-wdpc7\") pod \"dns-default-mtsz8\" (UID: \"1069a5ce-5720-4e0d-8bb8-cc874020e3dd\") " pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:24.646160 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.646144 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvg8b\" (UniqueName: \"kubernetes.io/projected/9b663748-9129-4d2c-b900-c5ddde8b4b63-kube-api-access-dvg8b\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:52:24.646640 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.646619 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrkbx\" (UniqueName: \"kubernetes.io/projected/09b6fe90-3962-49ef-bc16-486d09469b22-kube-api-access-mrkbx\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:24.646859 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.646842 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xdcg\" (UniqueName: \"kubernetes.io/projected/31e0a09f-ff45-482d-afec-c57bd2d41cb4-kube-api-access-9xdcg\") pod \"openshift-state-metrics-7bd7489957-pxkqg\" (UID: \"31e0a09f-ff45-482d-afec-c57bd2d41cb4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pxkqg" Oct 30 12:52:24.646937 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.646864 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flgvz\" (UniqueName: \"kubernetes.io/projected/5b885e3c-593b-45d1-b4d4-4604ca511662-kube-api-access-flgvz\") pod \"metrics-server-56b76dcb6d-fm86z\" (UID: \"5b885e3c-593b-45d1-b4d4-4604ca511662\") " pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:24.646937 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.646873 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9dqr\" (UniqueName: \"kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-kube-api-access-k9dqr\") pod \"prometheus-k8s-0\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:24.647160 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.647143 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5q2d\" (UniqueName: \"kubernetes.io/projected/83545835-dd10-4f8b-8015-25c579bdac3a-kube-api-access-w5q2d\") pod \"iptables-alerter-r9nqr\" (UID: \"83545835-dd10-4f8b-8015-25c579bdac3a\") " pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:52:24.647327 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.647311 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7llpr\" (UniqueName: \"kubernetes.io/projected/b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1-kube-api-access-7llpr\") pod \"kube-state-metrics-5dc8d7d89b-q2bdf\" (UID: \"b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q2bdf" Oct 30 12:52:24.648874 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.648855 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdp4l\" (UniqueName: \"kubernetes.io/projected/bb39553b-ccf2-4cb0-af1c-339373ec1b13-kube-api-access-bdp4l\") pod \"monitoring-plugin-77f4f76594-g2zfl\" (UID: \"bb39553b-ccf2-4cb0-af1c-339373ec1b13\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" Oct 30 12:52:24.650218 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.650201 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2r9g\" (UniqueName: \"kubernetes.io/projected/8915de45-eb8f-451b-8011-2cbf588de9b5-kube-api-access-n2r9g\") pod \"thanos-querier-79ff6956fb-ljgd2\" (UID: \"8915de45-eb8f-451b-8011-2cbf588de9b5\") " pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:24.651637 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.651623 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2wdq\" (UniqueName: \"kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-kube-api-access-h2wdq\") pod \"alertmanager-main-0\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:24.652033 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.652017 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c62x8\" (UniqueName: \"kubernetes.io/projected/fce77b3b-df3e-4d04-945c-68dc11edc61f-kube-api-access-c62x8\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.653327 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.653312 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gqh8\" (UniqueName: \"kubernetes.io/projected/109981ff-df1e-4279-bc7a-1e2b09f30ad1-kube-api-access-2gqh8\") pod \"node-resolver-qldmf\" (UID: \"109981ff-df1e-4279-bc7a-1e2b09f30ad1\") " pod="openshift-dns/node-resolver-qldmf" Oct 30 12:52:24.711738 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711655 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-sys\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.711738 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711708 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-socket-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.711738 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711721 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-sys\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.711738 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711734 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysconfig\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.712029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711767 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysctl-conf\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.712029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711796 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysconfig\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.712029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711797 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-kubelet-config\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:52:24.712029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711838 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-cni-bin\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.712029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711861 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-ovn\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.712029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711887 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-root\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.712029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711909 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-ovn\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.712029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711911 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-kubelet\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.712029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711860 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-socket-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.712029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711903 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-kubelet-config\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:52:24.712029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711943 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-cni-bin\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.712029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711959 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-root\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.712029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711964 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysctl-conf\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.712029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.711977 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-systemd\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.712029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712005 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-systemd\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.712029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712002 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-kubelet\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.712029 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712028 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-cni-netd\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712058 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-registration-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712101 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-modprobe-d\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712126 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09b6fe90-3962-49ef-bc16-486d09469b22-crio-socket\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712149 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-registration-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712101 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-cni-netd\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712154 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-kubelet\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712193 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-system-cni-dir\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712206 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-modprobe-d\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712218 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-hostroot\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712193 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-kubelet\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712231 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09b6fe90-3962-49ef-bc16-486d09469b22-crio-socket\") pod \"insights-runtime-extractor-4hts2\" (UID: \"09b6fe90-3962-49ef-bc16-486d09469b22\") " pod="openshift-insights/insights-runtime-extractor-4hts2" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712250 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-system-cni-dir\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712260 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-cnibin\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712283 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-run\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712283 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-hostroot\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712304 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-host\") pod \"node-ca-bm8bc\" (UID: \"8bc2e7c1-04ef-439a-90e5-57c4c200c62f\") " pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712364 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bc2e7c1-04ef-439a-90e5-57c4c200c62f-host\") pod \"node-ca-bm8bc\" (UID: \"8bc2e7c1-04ef-439a-90e5-57c4c200c62f\") " pod="openshift-image-registry/node-ca-bm8bc" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712386 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-run\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712422 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-kubelet-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712441 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-cnibin\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712474 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712528 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-kubelet-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712534 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-node-log\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712550 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-node-log\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712554 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-run-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712574 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-sys\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712597 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-sys-fs\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712637 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-run-netns\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712641 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-sys\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712662 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-netns\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.712665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712680 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-sys-fs\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712721 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-netns\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712718 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-device-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712755 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-host\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712770 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-dbus\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712785 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-system-cni-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712803 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83545835-dd10-4f8b-8015-25c579bdac3a-host-slash\") pod \"iptables-alerter-r9nqr\" (UID: \"83545835-dd10-4f8b-8015-25c579bdac3a\") " pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712824 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-cni-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712836 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-host\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712847 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-slash\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712851 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b497ad2-5bd4-4b7b-881d-d94c15fda49c-dbus\") pod \"global-pull-secret-syncer-2xg75\" (UID: \"0b497ad2-5bd4-4b7b-881d-d94c15fda49c\") " pod="kube-system/global-pull-secret-syncer-2xg75" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712856 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83545835-dd10-4f8b-8015-25c579bdac3a-host-slash\") pod \"iptables-alerter-r9nqr\" (UID: \"83545835-dd10-4f8b-8015-25c579bdac3a\") " pod="openshift-network-operator/iptables-alerter-r9nqr" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712767 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-run-netns\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712896 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-slash\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712925 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-cni-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712926 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-run-ovn-kubernetes\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712956 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-run-ovn-kubernetes\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712959 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-wtmp\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712957 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-system-cni-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712998 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713010 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60a88a61-3c25-48c8-84c5-b35bab4a058a-node-exporter-wtmp\") pod \"node-exporter-vdft6\" (UID: \"60a88a61-3c25-48c8-84c5-b35bab4a058a\") " pod="openshift-monitoring/node-exporter-vdft6" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713025 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-var-lib-kubelet\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713050 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-cnibin\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713062 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713081 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713102 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-var-lib-kubelet\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713113 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-cnibin\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713139 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-log-socket\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713167 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-multus-certs\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713195 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-etc-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713200 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-log-socket\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713236 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysctl-d\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713276 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-multus-certs\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713291 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-kubernetes\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713322 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-etc-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713230 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b663748-9129-4d2c-b900-c5ddde8b4b63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4mmzs\" (UID: \"9b663748-9129-4d2c-b900-c5ddde8b4b63\") " pod="openshift-multus/multus-additional-cni-plugins-4mmzs" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713335 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-socket-dir-parent\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713360 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-sysctl-d\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.712893 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-device-dir\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713390 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-kubernetes\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713383 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-cni-multus\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713425 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-etc-kubernetes\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713430 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-socket-dir-parent\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713435 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-var-lib-cni-multus\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713476 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-etc-kubernetes\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713477 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-systemd-units\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713549 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/109981ff-df1e-4279-bc7a-1e2b09f30ad1-hosts-file\") pod \"node-resolver-qldmf\" (UID: \"109981ff-df1e-4279-bc7a-1e2b09f30ad1\") " pod="openshift-dns/node-resolver-qldmf" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713576 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-conf-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713593 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-systemd-units\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713612 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-etc-selinux\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713641 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-cni-bin\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713647 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/109981ff-df1e-4279-bc7a-1e2b09f30ad1-hosts-file\") pod \"node-resolver-qldmf\" (UID: \"109981ff-df1e-4279-bc7a-1e2b09f30ad1\") " pod="openshift-dns/node-resolver-qldmf" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713665 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-k8s-cni-cncf-io\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713687 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-var-lib-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713687 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/404bb84d-f044-488e-a221-389f71e61181-etc-selinux\") pod \"aws-ebs-csi-driver-node-czl9k\" (UID: \"404bb84d-f044-488e-a221-389f71e61181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czl9k" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713717 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-systemd\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713751 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-lib-modules\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713794 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-host-run-k8s-cni-cncf-io\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713746 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94b7ca3d-22f2-4465-81a7-3a2d4082a6e9-multus-conf-dir\") pod \"multus-4tbwx\" (UID: \"94b7ca3d-22f2-4465-81a7-3a2d4082a6e9\") " pod="openshift-multus/multus-4tbwx" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713871 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-etc-systemd\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713910 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-host-cni-bin\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713912 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fce77b3b-df3e-4d04-945c-68dc11edc61f-var-lib-openvswitch\") pod \"ovnkube-node-jz7rd\" (UID: \"fce77b3b-df3e-4d04-945c-68dc11edc61f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:24.713932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.713930 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd-lib-modules\") pod \"tuned-bsmf4\" (UID: \"2fe4fa10-9b21-4d40-ae05-13bc0cfb67dd\") " pod="openshift-cluster-node-tuning-operator/tuned-bsmf4" Oct 30 12:52:24.888356 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:24.888329 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:25.248050 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:25.248017 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mtsz8" Oct 30 12:52:26.516247 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:26.516221 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:26.854223 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:26.854142 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-g2zfl" Oct 30 12:52:27.103318 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:27.103285 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:27.103557 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:27.103543 7010 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 12:52:28.112407 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:28.112379 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-79ff6956fb-ljgd2" Oct 30 12:52:28.419856 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:28.419825 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jz7rd" Oct 30 12:52:29.396834 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:29.396797 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 12:52:31.026889 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:31.026848 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:52:40.463014 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:40.462942 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zchsd" Oct 30 12:52:41.263142 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:41.263108 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-56b76dcb6d-fm86z" Oct 30 12:52:47.339452 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:47.339416 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:47.340160 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:47.339947 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="alertmanager" containerID="cri-o://f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70" gracePeriod=120 Oct 30 12:52:47.340160 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:47.339947 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="kube-rbac-proxy-metric" containerID="cri-o://27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde" gracePeriod=120 Oct 30 12:52:47.340160 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:47.340065 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="kube-rbac-proxy-web" containerID="cri-o://3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d" gracePeriod=120 Oct 30 12:52:47.340160 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:47.340051 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="prom-label-proxy" containerID="cri-o://adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08" gracePeriod=120 Oct 30 12:52:47.340721 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:47.340166 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="kube-rbac-proxy" containerID="cri-o://897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56" gracePeriod=120 Oct 30 12:52:47.340721 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:47.340055 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="config-reloader" containerID="cri-o://9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc" gracePeriod=120 Oct 30 12:52:47.649077 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:47.649046 7010 generic.go:356] "Generic (PLEG): container finished" podID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerID="adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08" exitCode=0 Oct 30 12:52:47.649077 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:47.649072 7010 generic.go:356] "Generic (PLEG): container finished" podID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerID="897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56" exitCode=0 Oct 30 12:52:47.649077 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:47.649081 7010 generic.go:356] "Generic (PLEG): container finished" podID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerID="9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc" exitCode=0 Oct 30 12:52:47.649077 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:47.649088 7010 generic.go:356] "Generic (PLEG): container finished" podID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerID="f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70" exitCode=0 Oct 30 12:52:47.649344 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:47.649123 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269","Type":"ContainerDied","Data":"adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08"} Oct 30 12:52:47.649344 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:47.649194 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269","Type":"ContainerDied","Data":"897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56"} Oct 30 12:52:47.649344 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:47.649222 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269","Type":"ContainerDied","Data":"9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc"} Oct 30 12:52:47.649344 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:47.649246 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269","Type":"ContainerDied","Data":"f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70"} Oct 30 12:52:48.587739 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.587710 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.620073 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.620036 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-metrics-client-ca\") pod \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " Oct 30 12:52:48.620073 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.620080 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-tls-assets\" (UniqueName: \"kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-tls-assets\") pod \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " Oct 30 12:52:48.620321 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.620109 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-main-tls\") pod \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " Oct 30 12:52:48.620321 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.620131 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-metric\") pod \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " Oct 30 12:52:48.620321 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.620153 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-volume\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-volume\") pod \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " Oct 30 12:52:48.620321 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.620175 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-main-db\") pod \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " Oct 30 12:52:48.620321 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.620192 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-web-config\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-web-config\") pod \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " Oct 30 12:52:48.620321 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.620226 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-out\") pod \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " Oct 30 12:52:48.620321 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.620262 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-web\") pod \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " Oct 30 12:52:48.620321 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.620310 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-cluster-tls-config\") pod \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " Oct 30 12:52:48.620746 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.620343 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-trusted-ca-bundle\") pod \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " Oct 30 12:52:48.620746 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.620375 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy\") pod \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " Oct 30 12:52:48.620746 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.620433 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-kube-api-access-h2wdq\" (UniqueName: \"kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-kube-api-access-h2wdq\") pod \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\" (UID: \"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269\") " Oct 30 12:52:48.620746 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.620639 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-metrics-client-ca" pod "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" (UID: "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:48.620964 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.620749 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-metrics-client-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.620964 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.620856 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-main-db" pod "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" (UID: "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:52:48.621572 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.621265 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-trusted-ca-bundle" pod "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" (UID: "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:48.623440 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.623402 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-kube-api-access-h2wdq" pod "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" (UID: "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269"). InnerVolumeSpecName "kube-api-access-h2wdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:48.623565 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.623542 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-out" pod "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" (UID: "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:52:48.623666 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.623631 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-tls-assets" pod "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" (UID: "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:48.623666 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.623657 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-volume" pod "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" (UID: "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:48.623833 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.623780 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-metric" pod "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" (UID: "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:48.623884 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.623864 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-main-tls" pod "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" (UID: "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:48.623984 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.623967 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-web" pod "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" (UID: "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:48.624543 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.624526 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy" pod "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" (UID: "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:48.630422 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.630389 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-cluster-tls-config" pod "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" (UID: "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:48.635714 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.635684 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-web-config" pod "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" (UID: "2f20c3e2-7ab8-4b88-9e8e-5e59023f1269"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:48.656155 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.656077 7010 generic.go:356] "Generic (PLEG): container finished" podID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerID="27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde" exitCode=0 Oct 30 12:52:48.656155 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.656102 7010 generic.go:356] "Generic (PLEG): container finished" podID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerID="3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d" exitCode=0 Oct 30 12:52:48.656325 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.656159 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.656325 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.656157 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269","Type":"ContainerDied","Data":"27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde"} Oct 30 12:52:48.656325 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.656226 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269","Type":"ContainerDied","Data":"3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d"} Oct 30 12:52:48.656325 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.656255 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f20c3e2-7ab8-4b88-9e8e-5e59023f1269","Type":"ContainerDied","Data":"9b05c2ce173dda96624c02c816fa0f7b5b5baeca0930eb0255007fd6da3fe6dd"} Oct 30 12:52:48.656325 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.656285 7010 scope.go:122] "RemoveContainer" containerID="adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08" Oct 30 12:52:48.664087 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.664063 7010 scope.go:122] "RemoveContainer" containerID="27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde" Oct 30 12:52:48.671609 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.671590 7010 scope.go:122] "RemoveContainer" containerID="897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56" Oct 30 12:52:48.681895 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.681871 7010 scope.go:122] "RemoveContainer" containerID="3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d" Oct 30 12:52:48.690957 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.690936 7010 scope.go:122] "RemoveContainer" containerID="9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc" Oct 30 12:52:48.700295 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.700268 7010 scope.go:122] "RemoveContainer" containerID="f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70" Oct 30 12:52:48.709533 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.709506 7010 scope.go:122] "RemoveContainer" containerID="9c17b806405334125a9c015df1ccb6aaf991fcea9c980a47541152a5fd6c8908" Oct 30 12:52:48.715045 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.715015 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:48.718471 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.718448 7010 scope.go:122] "RemoveContainer" containerID="adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08" Oct 30 12:52:48.718851 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:48.718828 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08\": container with ID starting with adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08 not found: ID does not exist" containerID="adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08" Oct 30 12:52:48.718912 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.718866 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08"} err="failed to get container status \"adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08\": rpc error: code = NotFound desc = could not find container \"adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08\": container with ID starting with adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08 not found: ID does not exist" Oct 30 12:52:48.718912 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.718902 7010 scope.go:122] "RemoveContainer" containerID="27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde" Oct 30 12:52:48.719222 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:48.719196 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde\": container with ID starting with 27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde not found: ID does not exist" containerID="27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde" Oct 30 12:52:48.719309 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.719227 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde"} err="failed to get container status \"27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde\": rpc error: code = NotFound desc = could not find container \"27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde\": container with ID starting with 27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde not found: ID does not exist" Oct 30 12:52:48.719309 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.719252 7010 scope.go:122] "RemoveContainer" containerID="897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56" Oct 30 12:52:48.719588 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:48.719567 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56\": container with ID starting with 897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56 not found: ID does not exist" containerID="897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56" Oct 30 12:52:48.719647 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.719597 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56"} err="failed to get container status \"897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56\": rpc error: code = NotFound desc = could not find container \"897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56\": container with ID starting with 897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56 not found: ID does not exist" Oct 30 12:52:48.719647 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.719617 7010 scope.go:122] "RemoveContainer" containerID="3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d" Oct 30 12:52:48.719924 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:48.719905 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d\": container with ID starting with 3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d not found: ID does not exist" containerID="3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d" Oct 30 12:52:48.719924 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.719926 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d"} err="failed to get container status \"3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d\": rpc error: code = NotFound desc = could not find container \"3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d\": container with ID starting with 3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d not found: ID does not exist" Oct 30 12:52:48.720040 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.719940 7010 scope.go:122] "RemoveContainer" containerID="9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc" Oct 30 12:52:48.720222 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:48.720192 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc\": container with ID starting with 9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc not found: ID does not exist" containerID="9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc" Oct 30 12:52:48.720351 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.720227 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc"} err="failed to get container status \"9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc\": rpc error: code = NotFound desc = could not find container \"9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc\": container with ID starting with 9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc not found: ID does not exist" Oct 30 12:52:48.720351 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.720250 7010 scope.go:122] "RemoveContainer" containerID="f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70" Oct 30 12:52:48.720544 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:48.720525 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70\": container with ID starting with f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70 not found: ID does not exist" containerID="f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70" Oct 30 12:52:48.720641 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.720549 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70"} err="failed to get container status \"f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70\": rpc error: code = NotFound desc = could not find container \"f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70\": container with ID starting with f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70 not found: ID does not exist" Oct 30 12:52:48.720641 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.720564 7010 scope.go:122] "RemoveContainer" containerID="9c17b806405334125a9c015df1ccb6aaf991fcea9c980a47541152a5fd6c8908" Oct 30 12:52:48.720861 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:48.720838 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c17b806405334125a9c015df1ccb6aaf991fcea9c980a47541152a5fd6c8908\": container with ID starting with 9c17b806405334125a9c015df1ccb6aaf991fcea9c980a47541152a5fd6c8908 not found: ID does not exist" containerID="9c17b806405334125a9c015df1ccb6aaf991fcea9c980a47541152a5fd6c8908" Oct 30 12:52:48.720949 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.720867 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c17b806405334125a9c015df1ccb6aaf991fcea9c980a47541152a5fd6c8908"} err="failed to get container status \"9c17b806405334125a9c015df1ccb6aaf991fcea9c980a47541152a5fd6c8908\": rpc error: code = NotFound desc = could not find container \"9c17b806405334125a9c015df1ccb6aaf991fcea9c980a47541152a5fd6c8908\": container with ID starting with 9c17b806405334125a9c015df1ccb6aaf991fcea9c980a47541152a5fd6c8908 not found: ID does not exist" Oct 30 12:52:48.720949 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.720887 7010 scope.go:122] "RemoveContainer" containerID="adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08" Oct 30 12:52:48.721155 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.721138 7010 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-tls-assets\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.721228 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.721154 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:48.721228 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.721158 7010 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-main-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.721228 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.721215 7010 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.721228 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.721228 7010 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.721507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.721239 7010 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-main-db\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.721507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.721249 7010 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-web-config\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.721507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.721257 7010 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-config-out\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.721507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.721267 7010 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.721507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.721276 7010 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-cluster-tls-config\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.721507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.721286 7010 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.721507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.721296 7010 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.721507 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.721308 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2wdq\" (UniqueName: \"kubernetes.io/projected/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269-kube-api-access-h2wdq\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:48.721805 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.721525 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08"} err="failed to get container status \"adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08\": rpc error: code = NotFound desc = could not find container \"adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08\": container with ID starting with adb67c6a253848b8ae7190d90af3cdf991c9b8446ed84a15c5df806b9d95fd08 not found: ID does not exist" Oct 30 12:52:48.721805 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.721546 7010 scope.go:122] "RemoveContainer" containerID="27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde" Oct 30 12:52:48.721873 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.721815 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde"} err="failed to get container status \"27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde\": rpc error: code = NotFound desc = could not find container \"27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde\": container with ID starting with 27dbfbc4dd6196ab717b0f10a6797ce4bedea7abfe0e57d3a1e600c154280bde not found: ID does not exist" Oct 30 12:52:48.721873 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.721831 7010 scope.go:122] "RemoveContainer" containerID="897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56" Oct 30 12:52:48.722094 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.722073 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56"} err="failed to get container status \"897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56\": rpc error: code = NotFound desc = could not find container \"897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56\": container with ID starting with 897860c07e63cd47f1fb1f498c2527f9252db941a809b8e3f71c5700784b4c56 not found: ID does not exist" Oct 30 12:52:48.722094 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.722091 7010 scope.go:122] "RemoveContainer" containerID="3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d" Oct 30 12:52:48.722337 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.722318 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d"} err="failed to get container status \"3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d\": rpc error: code = NotFound desc = could not find container \"3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d\": container with ID starting with 3bd1508940398959e2108e38b86b07c5e304e16a1d84d0f744dd9c9773a1a93d not found: ID does not exist" Oct 30 12:52:48.722382 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.722338 7010 scope.go:122] "RemoveContainer" containerID="9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc" Oct 30 12:52:48.722583 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.722565 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc"} err="failed to get container status \"9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc\": rpc error: code = NotFound desc = could not find container \"9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc\": container with ID starting with 9bd705aadf27dec1a3b424537ac9d5482b8f207e263ca9368e0b0da354a7a8fc not found: ID does not exist" Oct 30 12:52:48.722643 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.722583 7010 scope.go:122] "RemoveContainer" containerID="f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70" Oct 30 12:52:48.722819 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.722801 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70"} err="failed to get container status \"f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70\": rpc error: code = NotFound desc = could not find container \"f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70\": container with ID starting with f96d0161e3ef10612549966bf93e627d9c737271b551d218163a586bf33acd70 not found: ID does not exist" Oct 30 12:52:48.722892 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.722821 7010 scope.go:122] "RemoveContainer" containerID="9c17b806405334125a9c015df1ccb6aaf991fcea9c980a47541152a5fd6c8908" Oct 30 12:52:48.723128 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.723102 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c17b806405334125a9c015df1ccb6aaf991fcea9c980a47541152a5fd6c8908"} err="failed to get container status \"9c17b806405334125a9c015df1ccb6aaf991fcea9c980a47541152a5fd6c8908\": rpc error: code = NotFound desc = could not find container \"9c17b806405334125a9c015df1ccb6aaf991fcea9c980a47541152a5fd6c8908\": container with ID starting with 9c17b806405334125a9c015df1ccb6aaf991fcea9c980a47541152a5fd6c8908 not found: ID does not exist" Oct 30 12:52:48.765719 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.765684 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:48.766078 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766040 7010 cpu_manager.go:414] "containerMap: removing container" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="kube-rbac-proxy" Oct 30 12:52:48.766078 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766073 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="kube-rbac-proxy" Oct 30 12:52:48.766224 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766091 7010 cpu_manager.go:414] "containerMap: removing container" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="kube-rbac-proxy-metric" Oct 30 12:52:48.766224 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766101 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="kube-rbac-proxy-metric" Oct 30 12:52:48.766224 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766111 7010 cpu_manager.go:414] "containerMap: removing container" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="kube-rbac-proxy-web" Oct 30 12:52:48.766224 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766117 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="kube-rbac-proxy-web" Oct 30 12:52:48.766224 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766129 7010 cpu_manager.go:414] "containerMap: removing container" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="config-reloader" Oct 30 12:52:48.766224 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766134 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="config-reloader" Oct 30 12:52:48.766224 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766157 7010 cpu_manager.go:414] "containerMap: removing container" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="init-config-reloader" Oct 30 12:52:48.766224 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766163 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="init-config-reloader" Oct 30 12:52:48.766224 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766171 7010 cpu_manager.go:414] "containerMap: removing container" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="prom-label-proxy" Oct 30 12:52:48.766224 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766175 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="prom-label-proxy" Oct 30 12:52:48.766224 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766181 7010 cpu_manager.go:414] "containerMap: removing container" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="alertmanager" Oct 30 12:52:48.766224 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766185 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="alertmanager" Oct 30 12:52:48.766594 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766244 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="kube-rbac-proxy-web" Oct 30 12:52:48.766594 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766254 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="config-reloader" Oct 30 12:52:48.766594 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766261 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="init-config-reloader" Oct 30 12:52:48.766594 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766267 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="prom-label-proxy" Oct 30 12:52:48.766594 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766273 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="alertmanager" Oct 30 12:52:48.766594 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766283 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="kube-rbac-proxy" Oct 30 12:52:48.766594 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.766291 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" containerName="kube-rbac-proxy-metric" Oct 30 12:52:48.770007 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.769989 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.789693 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.789653 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 12:52:48.789863 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.789733 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 12:52:48.789863 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.789732 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 12:52:48.789863 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.789798 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 12:52:48.789863 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.789825 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 12:52:48.789863 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.789838 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 12:52:48.790062 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.789871 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 12:52:48.790062 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.789891 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 12:52:48.790062 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.789948 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6lqrk\"" Oct 30 12:52:48.801580 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.801547 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 12:52:48.822411 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.822374 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-config-volume\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.822411 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.822411 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.822649 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.822435 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbtlx\" (UniqueName: \"kubernetes.io/projected/143b92f5-8e39-431f-aaef-edcbc459acf5-kube-api-access-lbtlx\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.822649 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.822586 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.822649 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.822614 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.822742 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.822675 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/143b92f5-8e39-431f-aaef-edcbc459acf5-config-out\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.822742 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.822720 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.822805 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.822761 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/143b92f5-8e39-431f-aaef-edcbc459acf5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.822805 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.822796 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/143b92f5-8e39-431f-aaef-edcbc459acf5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.822866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.822830 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/143b92f5-8e39-431f-aaef-edcbc459acf5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.822866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.822845 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-web-config\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.822866 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.822859 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.822958 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.822902 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/143b92f5-8e39-431f-aaef-edcbc459acf5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.826343 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.826315 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:48.924067 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.923974 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.924067 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.924020 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.924067 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.924047 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/143b92f5-8e39-431f-aaef-edcbc459acf5-config-out\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.924067 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.924072 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.924400 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.924093 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/143b92f5-8e39-431f-aaef-edcbc459acf5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.924400 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.924121 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/143b92f5-8e39-431f-aaef-edcbc459acf5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.924400 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.924146 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/143b92f5-8e39-431f-aaef-edcbc459acf5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.924400 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.924166 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-web-config\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.924400 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.924188 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.924400 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.924216 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/143b92f5-8e39-431f-aaef-edcbc459acf5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.924400 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.924251 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-config-volume\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.924400 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.924273 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.924400 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.924302 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtlx\" (UniqueName: \"kubernetes.io/projected/143b92f5-8e39-431f-aaef-edcbc459acf5-kube-api-access-lbtlx\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.924890 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.924452 7010 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 12:52:48.925057 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.925029 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/143b92f5-8e39-431f-aaef-edcbc459acf5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.925290 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.925261 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/143b92f5-8e39-431f-aaef-edcbc459acf5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.925396 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.925333 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/143b92f5-8e39-431f-aaef-edcbc459acf5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.927707 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.927683 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-config-volume\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.927821 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.927723 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/143b92f5-8e39-431f-aaef-edcbc459acf5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.927931 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.927904 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.928035 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.927928 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.928035 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.927995 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.928157 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.928087 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/143b92f5-8e39-431f-aaef-edcbc459acf5-config-out\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.928292 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.928271 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-web-config\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.928405 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.928383 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.929020 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.929004 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/143b92f5-8e39-431f-aaef-edcbc459acf5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:48.954890 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:48.954860 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtlx\" (UniqueName: \"kubernetes.io/projected/143b92f5-8e39-431f-aaef-edcbc459acf5-kube-api-access-lbtlx\") pod \"alertmanager-main-0\" (UID: \"143b92f5-8e39-431f-aaef-edcbc459acf5\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:49.087650 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:49.087612 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 12:52:49.427889 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:49.427856 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 12:52:49.431146 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:49.431117 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143b92f5_8e39_431f_aaef_edcbc459acf5.slice/crio-1622cf9ca10e689756c73845f7080e19d1e437b52fd871c0be2bff3979b28d45 WatchSource:0}: Error finding container 1622cf9ca10e689756c73845f7080e19d1e437b52fd871c0be2bff3979b28d45: Status 404 returned error can't find the container with id 1622cf9ca10e689756c73845f7080e19d1e437b52fd871c0be2bff3979b28d45 Oct 30 12:52:49.485619 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:49.485592 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f20c3e2-7ab8-4b88-9e8e-5e59023f1269" path="/var/lib/kubelet/pods/2f20c3e2-7ab8-4b88-9e8e-5e59023f1269/volumes" Oct 30 12:52:49.667423 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:49.667331 7010 generic.go:356] "Generic (PLEG): container finished" podID="143b92f5-8e39-431f-aaef-edcbc459acf5" containerID="7ca2e143aded3954a25972b8a68b5308c0935197b860a9d05eeb934ac79e38f4" exitCode=0 Oct 30 12:52:49.667423 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:49.667371 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"143b92f5-8e39-431f-aaef-edcbc459acf5","Type":"ContainerDied","Data":"7ca2e143aded3954a25972b8a68b5308c0935197b860a9d05eeb934ac79e38f4"} Oct 30 12:52:49.667958 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:49.667448 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"143b92f5-8e39-431f-aaef-edcbc459acf5","Type":"ContainerStarted","Data":"1622cf9ca10e689756c73845f7080e19d1e437b52fd871c0be2bff3979b28d45"} Oct 30 12:52:50.674914 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:50.674830 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"143b92f5-8e39-431f-aaef-edcbc459acf5","Type":"ContainerStarted","Data":"e50d09155d0efd9e0bec33ec141f6acbf7e30e5da51523d48ef21a3b4f4768ea"} Oct 30 12:52:50.674914 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:50.674898 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"143b92f5-8e39-431f-aaef-edcbc459acf5","Type":"ContainerStarted","Data":"40602b51ed0a62a71131c4c7ab59dcde4f4f19e624303d9282322d0f671fe4ab"} Oct 30 12:52:50.675300 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:50.674923 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"143b92f5-8e39-431f-aaef-edcbc459acf5","Type":"ContainerStarted","Data":"40a4d140669597af0ff6067fa9d2927b5ac0101ad90e70b61a86e63382dad8bc"} Oct 30 12:52:50.675300 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:50.674949 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"143b92f5-8e39-431f-aaef-edcbc459acf5","Type":"ContainerStarted","Data":"14769bf897e5b0767d28ce644c9a2d54e9956403420b9a54aec6aa5899b598fb"} Oct 30 12:52:50.675300 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:50.674972 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"143b92f5-8e39-431f-aaef-edcbc459acf5","Type":"ContainerStarted","Data":"fdb7b657fa10ba3dc55f855ec2dc02912596a5394afdd4b9de482c79d09b4498"} Oct 30 12:52:50.675300 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:50.674995 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"143b92f5-8e39-431f-aaef-edcbc459acf5","Type":"ContainerStarted","Data":"ff785b0912a9257f80c6eae0ab98e3d38e25aa4287e527bc625d75f887c873b0"} Oct 30 12:52:50.768039 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:50.767979 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.754614639 podStartE2EDuration="2.767963591s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.348952ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:50.754924196 +0000 UTC m=+27.387111358" watchObservedRunningTime="2026-10-30 12:52:50.767963591 +0000 UTC m=+27.400150794" Oct 30 12:52:51.029219 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:51.029126 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dgnrj" Oct 30 12:52:51.902515 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:51.902469 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:51.903131 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:51.902861 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="config-reloader" containerID="cri-o://33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de" gracePeriod=600 Oct 30 12:52:51.903131 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:51.902882 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="prometheus" containerID="cri-o://9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a" gracePeriod=600 Oct 30 12:52:51.903131 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:51.902857 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="kube-rbac-proxy" containerID="cri-o://036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde" gracePeriod=600 Oct 30 12:52:51.903131 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:51.902862 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="thanos-sidecar" containerID="cri-o://d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12" gracePeriod=600 Oct 30 12:52:51.903131 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:51.902889 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="kube-rbac-proxy-web" containerID="cri-o://f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94" gracePeriod=600 Oct 30 12:52:51.903131 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:51.902904 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="kube-rbac-proxy-thanos" containerID="cri-o://df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b" gracePeriod=600 Oct 30 12:52:52.147783 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.147754 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.251604 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.251513 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-1\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.251604 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.251572 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-tls-assets\" (UniqueName: \"kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-tls-assets\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.251822 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.251668 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-config-out\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.251822 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.251705 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-0\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.251822 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.251727 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-2\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.251822 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.251758 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-metrics-client-certs\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.251822 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.251783 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-kube-api-access-k9dqr\" (UniqueName: \"kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-kube-api-access-k9dqr\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.251822 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.251799 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-tls\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.252129 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.251833 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.252129 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.251875 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-thanos-prometheus-http-client-file\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.252129 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.251905 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-grpc-tls\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.252129 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.251908 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-1" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:52.252129 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.251930 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-config\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-config\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.252129 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.251988 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-kubelet-serving-ca-bundle\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.252129 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.252037 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-db\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.252129 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.252070 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-web-config\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-web-config\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.252129 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.252112 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-metrics-client-ca\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.252557 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.252152 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.252557 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.252190 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-serving-certs-ca-bundle\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.252557 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.252218 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-trusted-ca-bundle\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.252557 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.252246 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-kube-rbac-proxy\") pod \"cd51364d-3838-4a3d-b470-ccf6483121e7\" (UID: \"cd51364d-3838-4a3d-b470-ccf6483121e7\") " Oct 30 12:52:52.252557 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.252431 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-2" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:52.252828 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.252562 7010 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.252828 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.252581 7010 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.253341 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.253313 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-0" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:52.253914 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.253887 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-db" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:52:52.255371 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.255122 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-tls-assets" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:52.255371 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.255199 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-metrics-client-certs" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:52.255566 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.255433 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-kube-rbac-proxy-web" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:52.255566 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.255458 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-kube-rbac-proxy" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:52.255760 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.255726 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-thanos-sidecar-tls" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:52.255869 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.255826 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-kubelet-serving-ca-bundle" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:52.255869 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.255844 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-serving-certs-ca-bundle" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:52.256313 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.256144 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-metrics-client-ca" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:52.256313 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.256255 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-config" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:52.256457 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.256360 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-trusted-ca-bundle" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:52.256457 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.256392 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-config-out" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:52:52.256936 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.256906 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-grpc-tls" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:52.257568 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.257539 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-kube-api-access-k9dqr" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "kube-api-access-k9dqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:52.258088 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.258066 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-tls" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:52.258618 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.258596 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-thanos-prometheus-http-client-file" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:52.268081 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.268046 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-web-config" pod "cd51364d-3838-4a3d-b470-ccf6483121e7" (UID: "cd51364d-3838-4a3d-b470-ccf6483121e7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:52:52.353628 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353581 7010 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-db\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.353628 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353616 7010 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-web-config\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.353628 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353627 7010 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-metrics-client-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.353628 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353639 7010 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.353859 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353649 7010 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.353859 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353658 7010 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.353859 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353667 7010 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-kube-rbac-proxy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.353859 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353677 7010 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-tls-assets\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.353859 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353685 7010 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd51364d-3838-4a3d-b470-ccf6483121e7-config-out\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.353859 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353692 7010 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.353859 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353701 7010 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-metrics-client-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.353859 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353709 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9dqr\" (UniqueName: \"kubernetes.io/projected/cd51364d-3838-4a3d-b470-ccf6483121e7-kube-api-access-k9dqr\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.353859 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353717 7010 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.353859 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353726 7010 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.353859 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353734 7010 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.353859 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353743 7010 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-secret-grpc-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.353859 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353751 7010 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd51364d-3838-4a3d-b470-ccf6483121e7-config\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.353859 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.353759 7010 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd51364d-3838-4a3d-b470-ccf6483121e7-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:52:52.685994 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.685954 7010 generic.go:356] "Generic (PLEG): container finished" podID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerID="df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b" exitCode=0 Oct 30 12:52:52.685994 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.685983 7010 generic.go:356] "Generic (PLEG): container finished" podID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerID="036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde" exitCode=0 Oct 30 12:52:52.685994 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.685991 7010 generic.go:356] "Generic (PLEG): container finished" podID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerID="f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94" exitCode=0 Oct 30 12:52:52.685994 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.685997 7010 generic.go:356] "Generic (PLEG): container finished" podID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerID="d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12" exitCode=0 Oct 30 12:52:52.685994 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.686002 7010 generic.go:356] "Generic (PLEG): container finished" podID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerID="33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de" exitCode=0 Oct 30 12:52:52.686392 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.686009 7010 generic.go:356] "Generic (PLEG): container finished" podID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerID="9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a" exitCode=0 Oct 30 12:52:52.686392 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.686036 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd51364d-3838-4a3d-b470-ccf6483121e7","Type":"ContainerDied","Data":"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b"} Oct 30 12:52:52.686392 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.686076 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.686392 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.686131 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd51364d-3838-4a3d-b470-ccf6483121e7","Type":"ContainerDied","Data":"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde"} Oct 30 12:52:52.686392 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.686162 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd51364d-3838-4a3d-b470-ccf6483121e7","Type":"ContainerDied","Data":"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94"} Oct 30 12:52:52.686392 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.686189 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd51364d-3838-4a3d-b470-ccf6483121e7","Type":"ContainerDied","Data":"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12"} Oct 30 12:52:52.686392 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.686202 7010 scope.go:122] "RemoveContainer" containerID="df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b" Oct 30 12:52:52.686392 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.686228 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd51364d-3838-4a3d-b470-ccf6483121e7","Type":"ContainerDied","Data":"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de"} Oct 30 12:52:52.686392 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.686271 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd51364d-3838-4a3d-b470-ccf6483121e7","Type":"ContainerDied","Data":"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a"} Oct 30 12:52:52.686392 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.686311 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd51364d-3838-4a3d-b470-ccf6483121e7","Type":"ContainerDied","Data":"cbad41beb16322a7113878ce13e65fa3aec8536102496e9412a71a69f9be6a4f"} Oct 30 12:52:52.694351 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.694331 7010 scope.go:122] "RemoveContainer" containerID="036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde" Oct 30 12:52:52.701918 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.701894 7010 scope.go:122] "RemoveContainer" containerID="f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94" Oct 30 12:52:52.708910 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.708883 7010 scope.go:122] "RemoveContainer" containerID="d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12" Oct 30 12:52:52.715906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.715888 7010 scope.go:122] "RemoveContainer" containerID="33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de" Oct 30 12:52:52.722981 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.722954 7010 scope.go:122] "RemoveContainer" containerID="9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a" Oct 30 12:52:52.730610 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.730588 7010 scope.go:122] "RemoveContainer" containerID="9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce" Oct 30 12:52:52.737892 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.737876 7010 scope.go:122] "RemoveContainer" containerID="df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b" Oct 30 12:52:52.738202 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:52.738183 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b\": container with ID starting with df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b not found: ID does not exist" containerID="df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b" Oct 30 12:52:52.738251 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.738213 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b"} err="failed to get container status \"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b\": rpc error: code = NotFound desc = could not find container \"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b\": container with ID starting with df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b not found: ID does not exist" Oct 30 12:52:52.738289 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.738251 7010 scope.go:122] "RemoveContainer" containerID="036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde" Oct 30 12:52:52.738481 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:52.738466 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde\": container with ID starting with 036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde not found: ID does not exist" containerID="036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde" Oct 30 12:52:52.738552 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.738505 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde"} err="failed to get container status \"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde\": rpc error: code = NotFound desc = could not find container \"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde\": container with ID starting with 036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde not found: ID does not exist" Oct 30 12:52:52.738552 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.738520 7010 scope.go:122] "RemoveContainer" containerID="f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94" Oct 30 12:52:52.738738 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:52.738720 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94\": container with ID starting with f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94 not found: ID does not exist" containerID="f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94" Oct 30 12:52:52.738777 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.738746 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94"} err="failed to get container status \"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94\": rpc error: code = NotFound desc = could not find container \"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94\": container with ID starting with f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94 not found: ID does not exist" Oct 30 12:52:52.738777 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.738764 7010 scope.go:122] "RemoveContainer" containerID="d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12" Oct 30 12:52:52.738974 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:52.738958 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12\": container with ID starting with d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12 not found: ID does not exist" containerID="d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12" Oct 30 12:52:52.739009 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.738980 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12"} err="failed to get container status \"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12\": rpc error: code = NotFound desc = could not find container \"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12\": container with ID starting with d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12 not found: ID does not exist" Oct 30 12:52:52.739009 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.738993 7010 scope.go:122] "RemoveContainer" containerID="33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de" Oct 30 12:52:52.739186 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:52.739172 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de\": container with ID starting with 33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de not found: ID does not exist" containerID="33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de" Oct 30 12:52:52.739222 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.739189 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de"} err="failed to get container status \"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de\": rpc error: code = NotFound desc = could not find container \"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de\": container with ID starting with 33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de not found: ID does not exist" Oct 30 12:52:52.739222 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.739202 7010 scope.go:122] "RemoveContainer" containerID="9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a" Oct 30 12:52:52.739386 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:52.739370 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a\": container with ID starting with 9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a not found: ID does not exist" containerID="9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a" Oct 30 12:52:52.739424 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.739389 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a"} err="failed to get container status \"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a\": rpc error: code = NotFound desc = could not find container \"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a\": container with ID starting with 9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a not found: ID does not exist" Oct 30 12:52:52.739424 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.739400 7010 scope.go:122] "RemoveContainer" containerID="9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce" Oct 30 12:52:52.739642 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:52.739626 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce\": container with ID starting with 9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce not found: ID does not exist" containerID="9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce" Oct 30 12:52:52.739687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.739659 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce"} err="failed to get container status \"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce\": rpc error: code = NotFound desc = could not find container \"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce\": container with ID starting with 9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce not found: ID does not exist" Oct 30 12:52:52.739687 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.739674 7010 scope.go:122] "RemoveContainer" containerID="df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b" Oct 30 12:52:52.739895 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.739879 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b"} err="failed to get container status \"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b\": rpc error: code = NotFound desc = could not find container \"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b\": container with ID starting with df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b not found: ID does not exist" Oct 30 12:52:52.739932 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.739895 7010 scope.go:122] "RemoveContainer" containerID="036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde" Oct 30 12:52:52.740132 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.740102 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde"} err="failed to get container status \"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde\": rpc error: code = NotFound desc = could not find container \"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde\": container with ID starting with 036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde not found: ID does not exist" Oct 30 12:52:52.740204 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.740133 7010 scope.go:122] "RemoveContainer" containerID="f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94" Oct 30 12:52:52.740386 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.740370 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94"} err="failed to get container status \"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94\": rpc error: code = NotFound desc = could not find container \"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94\": container with ID starting with f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94 not found: ID does not exist" Oct 30 12:52:52.740454 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.740387 7010 scope.go:122] "RemoveContainer" containerID="d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12" Oct 30 12:52:52.740635 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.740617 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12"} err="failed to get container status \"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12\": rpc error: code = NotFound desc = could not find container \"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12\": container with ID starting with d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12 not found: ID does not exist" Oct 30 12:52:52.740689 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.740636 7010 scope.go:122] "RemoveContainer" containerID="33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de" Oct 30 12:52:52.740858 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.740841 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de"} err="failed to get container status \"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de\": rpc error: code = NotFound desc = could not find container \"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de\": container with ID starting with 33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de not found: ID does not exist" Oct 30 12:52:52.740905 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.740858 7010 scope.go:122] "RemoveContainer" containerID="9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a" Oct 30 12:52:52.741116 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.741089 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a"} err="failed to get container status \"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a\": rpc error: code = NotFound desc = could not find container \"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a\": container with ID starting with 9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a not found: ID does not exist" Oct 30 12:52:52.741188 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.741116 7010 scope.go:122] "RemoveContainer" containerID="9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce" Oct 30 12:52:52.741333 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.741318 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce"} err="failed to get container status \"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce\": rpc error: code = NotFound desc = could not find container \"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce\": container with ID starting with 9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce not found: ID does not exist" Oct 30 12:52:52.741371 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.741332 7010 scope.go:122] "RemoveContainer" containerID="df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b" Oct 30 12:52:52.741551 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.741526 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b"} err="failed to get container status \"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b\": rpc error: code = NotFound desc = could not find container \"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b\": container with ID starting with df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b not found: ID does not exist" Oct 30 12:52:52.741551 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.741550 7010 scope.go:122] "RemoveContainer" containerID="036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde" Oct 30 12:52:52.741819 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.741803 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde"} err="failed to get container status \"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde\": rpc error: code = NotFound desc = could not find container \"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde\": container with ID starting with 036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde not found: ID does not exist" Oct 30 12:52:52.741819 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.741818 7010 scope.go:122] "RemoveContainer" containerID="f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94" Oct 30 12:52:52.742041 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.742025 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94"} err="failed to get container status \"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94\": rpc error: code = NotFound desc = could not find container \"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94\": container with ID starting with f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94 not found: ID does not exist" Oct 30 12:52:52.742103 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.742041 7010 scope.go:122] "RemoveContainer" containerID="d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12" Oct 30 12:52:52.742274 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.742251 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12"} err="failed to get container status \"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12\": rpc error: code = NotFound desc = could not find container \"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12\": container with ID starting with d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12 not found: ID does not exist" Oct 30 12:52:52.742344 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.742275 7010 scope.go:122] "RemoveContainer" containerID="33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de" Oct 30 12:52:52.742558 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.742539 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de"} err="failed to get container status \"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de\": rpc error: code = NotFound desc = could not find container \"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de\": container with ID starting with 33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de not found: ID does not exist" Oct 30 12:52:52.742614 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.742558 7010 scope.go:122] "RemoveContainer" containerID="9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a" Oct 30 12:52:52.742828 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.742796 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a"} err="failed to get container status \"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a\": rpc error: code = NotFound desc = could not find container \"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a\": container with ID starting with 9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a not found: ID does not exist" Oct 30 12:52:52.742828 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.742826 7010 scope.go:122] "RemoveContainer" containerID="9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce" Oct 30 12:52:52.743065 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.743048 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce"} err="failed to get container status \"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce\": rpc error: code = NotFound desc = could not find container \"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce\": container with ID starting with 9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce not found: ID does not exist" Oct 30 12:52:52.743112 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.743066 7010 scope.go:122] "RemoveContainer" containerID="df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b" Oct 30 12:52:52.743296 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.743275 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b"} err="failed to get container status \"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b\": rpc error: code = NotFound desc = could not find container \"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b\": container with ID starting with df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b not found: ID does not exist" Oct 30 12:52:52.743340 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.743297 7010 scope.go:122] "RemoveContainer" containerID="036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde" Oct 30 12:52:52.743545 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.743528 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde"} err="failed to get container status \"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde\": rpc error: code = NotFound desc = could not find container \"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde\": container with ID starting with 036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde not found: ID does not exist" Oct 30 12:52:52.743588 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.743545 7010 scope.go:122] "RemoveContainer" containerID="f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94" Oct 30 12:52:52.743758 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.743741 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94"} err="failed to get container status \"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94\": rpc error: code = NotFound desc = could not find container \"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94\": container with ID starting with f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94 not found: ID does not exist" Oct 30 12:52:52.743797 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.743758 7010 scope.go:122] "RemoveContainer" containerID="d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12" Oct 30 12:52:52.743986 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.743970 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12"} err="failed to get container status \"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12\": rpc error: code = NotFound desc = could not find container \"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12\": container with ID starting with d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12 not found: ID does not exist" Oct 30 12:52:52.744022 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.743987 7010 scope.go:122] "RemoveContainer" containerID="33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de" Oct 30 12:52:52.744183 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.744168 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de"} err="failed to get container status \"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de\": rpc error: code = NotFound desc = could not find container \"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de\": container with ID starting with 33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de not found: ID does not exist" Oct 30 12:52:52.744229 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.744183 7010 scope.go:122] "RemoveContainer" containerID="9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a" Oct 30 12:52:52.744368 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.744350 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a"} err="failed to get container status \"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a\": rpc error: code = NotFound desc = could not find container \"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a\": container with ID starting with 9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a not found: ID does not exist" Oct 30 12:52:52.744403 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.744369 7010 scope.go:122] "RemoveContainer" containerID="9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce" Oct 30 12:52:52.744590 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.744574 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce"} err="failed to get container status \"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce\": rpc error: code = NotFound desc = could not find container \"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce\": container with ID starting with 9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce not found: ID does not exist" Oct 30 12:52:52.744637 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.744590 7010 scope.go:122] "RemoveContainer" containerID="df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b" Oct 30 12:52:52.744790 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.744776 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b"} err="failed to get container status \"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b\": rpc error: code = NotFound desc = could not find container \"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b\": container with ID starting with df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b not found: ID does not exist" Oct 30 12:52:52.744831 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.744790 7010 scope.go:122] "RemoveContainer" containerID="036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde" Oct 30 12:52:52.744985 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.744970 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde"} err="failed to get container status \"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde\": rpc error: code = NotFound desc = could not find container \"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde\": container with ID starting with 036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde not found: ID does not exist" Oct 30 12:52:52.745027 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.744985 7010 scope.go:122] "RemoveContainer" containerID="f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94" Oct 30 12:52:52.745172 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.745157 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94"} err="failed to get container status \"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94\": rpc error: code = NotFound desc = could not find container \"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94\": container with ID starting with f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94 not found: ID does not exist" Oct 30 12:52:52.745217 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.745171 7010 scope.go:122] "RemoveContainer" containerID="d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12" Oct 30 12:52:52.745363 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.745346 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12"} err="failed to get container status \"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12\": rpc error: code = NotFound desc = could not find container \"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12\": container with ID starting with d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12 not found: ID does not exist" Oct 30 12:52:52.745408 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.745363 7010 scope.go:122] "RemoveContainer" containerID="33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de" Oct 30 12:52:52.745598 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.745582 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de"} err="failed to get container status \"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de\": rpc error: code = NotFound desc = could not find container \"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de\": container with ID starting with 33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de not found: ID does not exist" Oct 30 12:52:52.745642 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.745597 7010 scope.go:122] "RemoveContainer" containerID="9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a" Oct 30 12:52:52.745774 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.745759 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a"} err="failed to get container status \"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a\": rpc error: code = NotFound desc = could not find container \"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a\": container with ID starting with 9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a not found: ID does not exist" Oct 30 12:52:52.745817 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.745774 7010 scope.go:122] "RemoveContainer" containerID="9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce" Oct 30 12:52:52.745953 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.745939 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce"} err="failed to get container status \"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce\": rpc error: code = NotFound desc = could not find container \"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce\": container with ID starting with 9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce not found: ID does not exist" Oct 30 12:52:52.745996 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.745953 7010 scope.go:122] "RemoveContainer" containerID="df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b" Oct 30 12:52:52.746142 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.746124 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b"} err="failed to get container status \"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b\": rpc error: code = NotFound desc = could not find container \"df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b\": container with ID starting with df09e3c39a1e770e109df82ddbb4a06bb94b358af322a6e994864202be10ba5b not found: ID does not exist" Oct 30 12:52:52.746185 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.746144 7010 scope.go:122] "RemoveContainer" containerID="036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde" Oct 30 12:52:52.746336 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.746322 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde"} err="failed to get container status \"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde\": rpc error: code = NotFound desc = could not find container \"036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde\": container with ID starting with 036517acd7d84f97a2e359b477d936faa9193ccd64c3b2470e5a86722154dfde not found: ID does not exist" Oct 30 12:52:52.746372 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.746337 7010 scope.go:122] "RemoveContainer" containerID="f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94" Oct 30 12:52:52.746526 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.746511 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94"} err="failed to get container status \"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94\": rpc error: code = NotFound desc = could not find container \"f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94\": container with ID starting with f214d9e5c298a379f0d12b2964bc6e9f961dbe30898fa9cc3eb7cd6347994f94 not found: ID does not exist" Oct 30 12:52:52.746570 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.746526 7010 scope.go:122] "RemoveContainer" containerID="d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12" Oct 30 12:52:52.746734 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.746719 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12"} err="failed to get container status \"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12\": rpc error: code = NotFound desc = could not find container \"d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12\": container with ID starting with d7dac199260626cbb07bc8a9922de2c574f6444dd81c87ffdfeb5c43f3ccca12 not found: ID does not exist" Oct 30 12:52:52.746769 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.746734 7010 scope.go:122] "RemoveContainer" containerID="33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de" Oct 30 12:52:52.746919 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.746905 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de"} err="failed to get container status \"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de\": rpc error: code = NotFound desc = could not find container \"33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de\": container with ID starting with 33120de9dc56164461389a1375f03f3b1417c9d4f46192e1c78a55183068e0de not found: ID does not exist" Oct 30 12:52:52.746963 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.746919 7010 scope.go:122] "RemoveContainer" containerID="9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a" Oct 30 12:52:52.747102 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.747086 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a"} err="failed to get container status \"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a\": rpc error: code = NotFound desc = could not find container \"9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a\": container with ID starting with 9dca7226e74d1179f0319ef1bd03705f5cc841c9c35a298cbc51765db50c245a not found: ID does not exist" Oct 30 12:52:52.747146 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.747102 7010 scope.go:122] "RemoveContainer" containerID="9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce" Oct 30 12:52:52.747295 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.747278 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce"} err="failed to get container status \"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce\": rpc error: code = NotFound desc = could not find container \"9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce\": container with ID starting with 9afd1e97cfe69a6a30c5ff098c1282a688093222b53e17977ca835fa73ce01ce not found: ID does not exist" Oct 30 12:52:52.754447 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.754425 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:52.766828 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.766800 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:52.796633 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.796601 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:52.796912 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.796898 7010 cpu_manager.go:414] "containerMap: removing container" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="kube-rbac-proxy-web" Oct 30 12:52:52.796963 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.796916 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="kube-rbac-proxy-web" Oct 30 12:52:52.796963 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.796930 7010 cpu_manager.go:414] "containerMap: removing container" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="init-config-reloader" Oct 30 12:52:52.796963 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.796935 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="init-config-reloader" Oct 30 12:52:52.796963 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.796944 7010 cpu_manager.go:414] "containerMap: removing container" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="kube-rbac-proxy-thanos" Oct 30 12:52:52.796963 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.796949 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="kube-rbac-proxy-thanos" Oct 30 12:52:52.796963 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.796956 7010 cpu_manager.go:414] "containerMap: removing container" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="config-reloader" Oct 30 12:52:52.796963 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.796960 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="config-reloader" Oct 30 12:52:52.797152 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.796975 7010 cpu_manager.go:414] "containerMap: removing container" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="prometheus" Oct 30 12:52:52.797152 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.796981 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="prometheus" Oct 30 12:52:52.797152 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.796994 7010 cpu_manager.go:414] "containerMap: removing container" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="thanos-sidecar" Oct 30 12:52:52.797152 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.796999 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="thanos-sidecar" Oct 30 12:52:52.797152 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.797010 7010 cpu_manager.go:414] "containerMap: removing container" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="kube-rbac-proxy" Oct 30 12:52:52.797152 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.797014 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="kube-rbac-proxy" Oct 30 12:52:52.797152 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.797062 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="kube-rbac-proxy-thanos" Oct 30 12:52:52.797152 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.797071 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="config-reloader" Oct 30 12:52:52.797152 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.797078 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="prometheus" Oct 30 12:52:52.797152 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.797083 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="thanos-sidecar" Oct 30 12:52:52.797152 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.797089 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="kube-rbac-proxy" Oct 30 12:52:52.797152 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.797096 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="kube-rbac-proxy-web" Oct 30 12:52:52.797152 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.797101 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" containerName="init-config-reloader" Oct 30 12:52:52.801011 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.800994 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.820104 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.820082 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 12:52:52.820567 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.820551 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 12:52:52.820642 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.820624 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 12:52:52.820884 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.820862 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 12:52:52.821143 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.821116 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 12:52:52.821143 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.821117 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-27xf9\"" Oct 30 12:52:52.821257 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.821127 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 12:52:52.821257 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.821165 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ee888a1mr1r2m\"" Oct 30 12:52:52.821257 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.821191 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 12:52:52.821257 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.821217 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 12:52:52.821257 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.821224 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 12:52:52.821435 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.821327 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 12:52:52.821517 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.821481 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 12:52:52.832841 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.832815 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 12:52:52.834241 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.834215 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 12:52:52.834465 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.834438 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 12:52:52.835449 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.835426 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:52.857074 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.857045 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.857074 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.857082 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.857258 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.857155 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-config\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.857258 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.857233 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.857356 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.857281 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.857356 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.857316 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.857356 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.857348 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.857506 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.857376 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-web-config\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.857557 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.857507 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f97cce5e-e85c-470a-9cd1-4b3774e90600-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.857591 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.857574 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59b6k\" (UniqueName: \"kubernetes.io/projected/f97cce5e-e85c-470a-9cd1-4b3774e90600-kube-api-access-59b6k\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.857632 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.857615 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.857698 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.857679 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.857751 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.857716 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f97cce5e-e85c-470a-9cd1-4b3774e90600-config-out\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.857982 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.857953 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.858066 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.857992 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f97cce5e-e85c-470a-9cd1-4b3774e90600-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.858066 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.858046 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.858179 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.858154 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.858229 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.858186 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.858268 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.858218 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.858299 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.858275 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.959365 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959283 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f97cce5e-e85c-470a-9cd1-4b3774e90600-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.959365 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959323 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59b6k\" (UniqueName: \"kubernetes.io/projected/f97cce5e-e85c-470a-9cd1-4b3774e90600-kube-api-access-59b6k\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.959365 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959344 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.959365 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959363 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960021 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959511 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f97cce5e-e85c-470a-9cd1-4b3774e90600-config-out\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960021 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959564 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960021 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959597 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f97cce5e-e85c-470a-9cd1-4b3774e90600-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960021 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959618 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960021 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959656 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960021 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959682 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960021 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959712 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960021 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959739 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960021 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959785 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960021 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959822 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960021 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959861 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-config\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960021 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959903 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960021 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959926 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960021 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959955 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960021 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.959986 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960021 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.960010 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-web-config\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960840 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.960182 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960840 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.960276 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960840 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.960442 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f97cce5e-e85c-470a-9cd1-4b3774e90600-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.960840 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.960829 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.961284 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.961116 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.961856 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.961544 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.962454 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.962419 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f97cce5e-e85c-470a-9cd1-4b3774e90600-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.962924 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.962589 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.963552 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.963522 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.963650 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.963590 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-web-config\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.964116 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.964003 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f97cce5e-e85c-470a-9cd1-4b3774e90600-config-out\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.964116 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.964025 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.964116 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.964109 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f97cce5e-e85c-470a-9cd1-4b3774e90600-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.964337 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.964241 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.964790 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.964770 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-config\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.964880 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.964863 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.965187 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.965170 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.965966 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.965950 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.966552 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.966465 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f97cce5e-e85c-470a-9cd1-4b3774e90600-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:52.983408 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:52.983383 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59b6k\" (UniqueName: \"kubernetes.io/projected/f97cce5e-e85c-470a-9cd1-4b3774e90600-kube-api-access-59b6k\") pod \"prometheus-k8s-0\" (UID: \"f97cce5e-e85c-470a-9cd1-4b3774e90600\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:53.119672 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:53.119634 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:53.464053 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:53.464025 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 12:52:53.467415 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:52:53.467390 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97cce5e_e85c_470a_9cd1_4b3774e90600.slice/crio-531e1b370b22167dda593c22f4f60dea204a86b010ffaa70a678909963e4762f WatchSource:0}: Error finding container 531e1b370b22167dda593c22f4f60dea204a86b010ffaa70a678909963e4762f: Status 404 returned error can't find the container with id 531e1b370b22167dda593c22f4f60dea204a86b010ffaa70a678909963e4762f Oct 30 12:52:53.490600 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:53.490567 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd51364d-3838-4a3d-b470-ccf6483121e7" path="/var/lib/kubelet/pods/cd51364d-3838-4a3d-b470-ccf6483121e7/volumes" Oct 30 12:52:53.691920 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:53.691884 7010 generic.go:356] "Generic (PLEG): container finished" podID="f97cce5e-e85c-470a-9cd1-4b3774e90600" containerID="650650e8a2b41333318a8392a5583c6fc32f225d05fcb7b34860598b2805ab91" exitCode=0 Oct 30 12:52:53.692092 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:53.691923 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f97cce5e-e85c-470a-9cd1-4b3774e90600","Type":"ContainerDied","Data":"650650e8a2b41333318a8392a5583c6fc32f225d05fcb7b34860598b2805ab91"} Oct 30 12:52:53.692092 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:53.691977 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f97cce5e-e85c-470a-9cd1-4b3774e90600","Type":"ContainerStarted","Data":"531e1b370b22167dda593c22f4f60dea204a86b010ffaa70a678909963e4762f"} Oct 30 12:52:54.698919 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:54.698879 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f97cce5e-e85c-470a-9cd1-4b3774e90600","Type":"ContainerStarted","Data":"e06a3737fd861757ebaca4df5549d1ce00cefa52fb6b0b1aa6cece9a5b1fda8f"} Oct 30 12:52:54.699413 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:54.699005 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f97cce5e-e85c-470a-9cd1-4b3774e90600","Type":"ContainerStarted","Data":"095e09a406589cbaf1bfd2177f72519ee7beb22a1afd605eb16e443b177333f5"} Oct 30 12:52:54.699413 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:54.699049 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f97cce5e-e85c-470a-9cd1-4b3774e90600","Type":"ContainerStarted","Data":"1b599d86bcdfa6899dd264e34597bccfa98ace510bdb99a210d1caf7998f443e"} Oct 30 12:52:54.699413 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:54.699097 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f97cce5e-e85c-470a-9cd1-4b3774e90600","Type":"ContainerStarted","Data":"08da5cd26f620439c73683e7930df9e19e5cdf4f4c430d5ba8cc2a62cabf575f"} Oct 30 12:52:54.699413 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:54.699139 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f97cce5e-e85c-470a-9cd1-4b3774e90600","Type":"ContainerStarted","Data":"4c64f93e4227e913e125deceff51a3678a30c6110cb825e09b58ccb5e9f06928"} Oct 30 12:52:54.699413 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:54.699176 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f97cce5e-e85c-470a-9cd1-4b3774e90600","Type":"ContainerStarted","Data":"4bdafb9fc9fa9c017585bea5527cf65e2a1da360a471a96c4510afb3c86e302b"} Oct 30 12:52:54.769426 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:54.769359 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7526062959999997 podStartE2EDuration="2.76934688s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.740584ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:54.76874682 +0000 UTC m=+31.400934006" watchObservedRunningTime="2026-10-30 12:52:54.76934688 +0000 UTC m=+31.401534074" Oct 30 12:52:58.120249 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:58.120212 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:59.255943 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:59.255902 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:59.272020 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:59.271987 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:52:59.399408 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:52:59.399367 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 12:52:59.736021 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:52:59.735994 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 12:53:29.397706 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:53:29.397667 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 12:53:59.398228 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:53:59.398187 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 12:54:29.397847 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:54:29.397810 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 12:54:59.397148 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:54:59.397107 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 12:55:29.397712 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:55:29.397675 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 12:55:34.246275 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:34.246186 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-8l98b"] Oct 30 12:55:34.249356 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:34.249340 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-8l98b" Oct 30 12:55:34.266141 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:34.266118 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-58t8w\"" Oct 30 12:55:34.266500 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:34.266474 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 12:55:34.278971 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:34.278947 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 12:55:34.279791 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:34.279769 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-8l98b"] Oct 30 12:55:34.309483 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:34.309455 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wqth\" (UniqueName: \"kubernetes.io/projected/0aba3f2a-6a4d-4c1d-84f3-a72fc457c23b-kube-api-access-4wqth\") pod \"openshift-lws-operator-8465994787-8l98b\" (UID: \"0aba3f2a-6a4d-4c1d-84f3-a72fc457c23b\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-8l98b" Oct 30 12:55:34.309617 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:34.309499 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0aba3f2a-6a4d-4c1d-84f3-a72fc457c23b-tmp\") pod \"openshift-lws-operator-8465994787-8l98b\" (UID: \"0aba3f2a-6a4d-4c1d-84f3-a72fc457c23b\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-8l98b" Oct 30 12:55:34.410400 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:34.410365 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wqth\" (UniqueName: \"kubernetes.io/projected/0aba3f2a-6a4d-4c1d-84f3-a72fc457c23b-kube-api-access-4wqth\") pod \"openshift-lws-operator-8465994787-8l98b\" (UID: \"0aba3f2a-6a4d-4c1d-84f3-a72fc457c23b\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-8l98b" Oct 30 12:55:34.410400 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:34.410415 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0aba3f2a-6a4d-4c1d-84f3-a72fc457c23b-tmp\") pod \"openshift-lws-operator-8465994787-8l98b\" (UID: \"0aba3f2a-6a4d-4c1d-84f3-a72fc457c23b\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-8l98b" Oct 30 12:55:34.410765 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:34.410747 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0aba3f2a-6a4d-4c1d-84f3-a72fc457c23b-tmp\") pod \"openshift-lws-operator-8465994787-8l98b\" (UID: \"0aba3f2a-6a4d-4c1d-84f3-a72fc457c23b\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-8l98b" Oct 30 12:55:34.435511 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:34.435448 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wqth\" (UniqueName: \"kubernetes.io/projected/0aba3f2a-6a4d-4c1d-84f3-a72fc457c23b-kube-api-access-4wqth\") pod \"openshift-lws-operator-8465994787-8l98b\" (UID: \"0aba3f2a-6a4d-4c1d-84f3-a72fc457c23b\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-8l98b" Oct 30 12:55:34.573998 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:34.573926 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-8l98b" Oct 30 12:55:34.883432 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:34.883406 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-8l98b"] Oct 30 12:55:34.886829 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:55:34.886801 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0aba3f2a_6a4d_4c1d_84f3_a72fc457c23b.slice/crio-ec088e88b4ebb9acc29177bbe5bfa38607f3ebfd5921e4ac2a7b517e739b14c1 WatchSource:0}: Error finding container ec088e88b4ebb9acc29177bbe5bfa38607f3ebfd5921e4ac2a7b517e739b14c1: Status 404 returned error can't find the container with id ec088e88b4ebb9acc29177bbe5bfa38607f3ebfd5921e4ac2a7b517e739b14c1 Oct 30 12:55:34.888530 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:34.888510 7010 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:55:35.265663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:35.265582 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-8l98b" event={"ID":"0aba3f2a-6a4d-4c1d-84f3-a72fc457c23b","Type":"ContainerStarted","Data":"ec088e88b4ebb9acc29177bbe5bfa38607f3ebfd5921e4ac2a7b517e739b14c1"} Oct 30 12:55:38.279526 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:38.279469 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-8l98b" event={"ID":"0aba3f2a-6a4d-4c1d-84f3-a72fc457c23b","Type":"ContainerStarted","Data":"2a1035ccf8a741b13e7cc908dfbf271e8ef8e568f432e96be096153551473d7f"} Oct 30 12:55:38.322451 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:38.322401 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-8l98b" podStartSLOduration=1.576443182 podStartE2EDuration="4.322385858s" totalImagesPullingTime="2.745942676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:38.321830917 +0000 UTC m=+194.954018081" watchObservedRunningTime="2026-10-30 12:55:38.322385858 +0000 UTC m=+194.954573017" Oct 30 12:55:52.566584 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.566551 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4"] Oct 30 12:55:52.570819 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.570788 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.608713 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.608684 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4"] Oct 30 12:55:52.664807 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.664778 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 12:55:52.665741 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.665719 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 12:55:52.665869 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.665757 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 12:55:52.665869 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.665770 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lvqkv\"" Oct 30 12:55:52.665869 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.665770 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 12:55:52.666307 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.666294 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 12:55:52.667662 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.667642 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40b0026f-ae61-4211-b0f3-18e32fe5b20f-webhook-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.667737 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.667681 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/40b0026f-ae61-4211-b0f3-18e32fe5b20f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.667737 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.667698 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40b0026f-ae61-4211-b0f3-18e32fe5b20f-metrics-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.667819 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.667773 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/40b0026f-ae61-4211-b0f3-18e32fe5b20f-manifests\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.667819 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.667809 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prd6s\" (UniqueName: \"kubernetes.io/projected/40b0026f-ae61-4211-b0f3-18e32fe5b20f-kube-api-access-prd6s\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.667937 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.667922 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40b0026f-ae61-4211-b0f3-18e32fe5b20f-tmp\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.770518 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.768926 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40b0026f-ae61-4211-b0f3-18e32fe5b20f-tmp\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.770518 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.769016 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40b0026f-ae61-4211-b0f3-18e32fe5b20f-webhook-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.770518 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.769063 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/40b0026f-ae61-4211-b0f3-18e32fe5b20f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.770518 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.769089 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40b0026f-ae61-4211-b0f3-18e32fe5b20f-metrics-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.770518 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.769132 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/40b0026f-ae61-4211-b0f3-18e32fe5b20f-manifests\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.770518 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.769159 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prd6s\" (UniqueName: \"kubernetes.io/projected/40b0026f-ae61-4211-b0f3-18e32fe5b20f-kube-api-access-prd6s\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.773469 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.773413 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/40b0026f-ae61-4211-b0f3-18e32fe5b20f-manifests\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.773714 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.773688 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40b0026f-ae61-4211-b0f3-18e32fe5b20f-tmp\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.775803 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.775778 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40b0026f-ae61-4211-b0f3-18e32fe5b20f-metrics-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.775938 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.775915 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40b0026f-ae61-4211-b0f3-18e32fe5b20f-webhook-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.776733 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.776715 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/40b0026f-ae61-4211-b0f3-18e32fe5b20f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.793663 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.793642 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prd6s\" (UniqueName: \"kubernetes.io/projected/40b0026f-ae61-4211-b0f3-18e32fe5b20f-kube-api-access-prd6s\") pod \"opendatahub-operator-controller-manager-678d69fff5-m9kz4\" (UID: \"40b0026f-ae61-4211-b0f3-18e32fe5b20f\") " pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:52.884823 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:52.884794 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:53.312656 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:53.312569 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4"] Oct 30 12:55:53.330034 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:55:53.330004 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b0026f_ae61_4211_b0f3_18e32fe5b20f.slice/crio-c9c8d6fa0a0f6a7c14bd866ba72e8ebffd52ff587333313c80cd3ed646e9dd71 WatchSource:0}: Error finding container c9c8d6fa0a0f6a7c14bd866ba72e8ebffd52ff587333313c80cd3ed646e9dd71: Status 404 returned error can't find the container with id c9c8d6fa0a0f6a7c14bd866ba72e8ebffd52ff587333313c80cd3ed646e9dd71 Oct 30 12:55:54.344288 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:54.344182 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" event={"ID":"40b0026f-ae61-4211-b0f3-18e32fe5b20f","Type":"ContainerStarted","Data":"c9c8d6fa0a0f6a7c14bd866ba72e8ebffd52ff587333313c80cd3ed646e9dd71"} Oct 30 12:55:56.353860 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:56.353766 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" event={"ID":"40b0026f-ae61-4211-b0f3-18e32fe5b20f","Type":"ContainerStarted","Data":"aa3ce6b23b8f10913969928ca193e1ee701927b4761ac68c0e77083816d2d930"} Oct 30 12:55:57.358880 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:57.358843 7010 generic.go:356] "Generic (PLEG): container finished" podID="40b0026f-ae61-4211-b0f3-18e32fe5b20f" containerID="aa3ce6b23b8f10913969928ca193e1ee701927b4761ac68c0e77083816d2d930" exitCode=0 Oct 30 12:55:57.359310 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:57.358935 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" event={"ID":"40b0026f-ae61-4211-b0f3-18e32fe5b20f","Type":"ContainerDied","Data":"aa3ce6b23b8f10913969928ca193e1ee701927b4761ac68c0e77083816d2d930"} Oct 30 12:55:58.364509 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:58.364455 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" event={"ID":"40b0026f-ae61-4211-b0f3-18e32fe5b20f","Type":"ContainerStarted","Data":"b297aa8ef6109e4e36445e07bdfd45ddc51a99525208fc4104a15dac41450118"} Oct 30 12:55:58.364914 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:58.364565 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:55:58.455243 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:55:58.455189 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" podStartSLOduration=2.921978556 podStartE2EDuration="6.455175823s" totalImagesPullingTime="3.052621865s" totalInitContainerRuntime="480.575402ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:58.442189341 +0000 UTC m=+215.074376502" watchObservedRunningTime="2026-10-30 12:55:58.455175823 +0000 UTC m=+215.087363040" Oct 30 12:55:59.397308 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:55:59.397267 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 12:56:09.370262 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:09.370230 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-678d69fff5-m9kz4" Oct 30 12:56:12.460510 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.460458 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65dc4d65b9-cvtrj"] Oct 30 12:56:12.479557 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.479531 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-cvtrj" Oct 30 12:56:12.489396 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.489366 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65dc4d65b9-cvtrj"] Oct 30 12:56:12.576332 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.576300 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 12:56:12.576664 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.576646 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 12:56:12.577605 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.577588 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 12:56:12.578022 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.578007 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 12:56:12.578083 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.578021 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vz8qn\"" Oct 30 12:56:12.641079 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.641056 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9qpt\" (UniqueName: \"kubernetes.io/projected/ea775e41-0676-42f8-937c-5b771342e724-kube-api-access-l9qpt\") pod \"kube-auth-proxy-65dc4d65b9-cvtrj\" (UID: \"ea775e41-0676-42f8-937c-5b771342e724\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-cvtrj" Oct 30 12:56:12.641237 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.641085 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea775e41-0676-42f8-937c-5b771342e724-tls-certs\") pod \"kube-auth-proxy-65dc4d65b9-cvtrj\" (UID: \"ea775e41-0676-42f8-937c-5b771342e724\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-cvtrj" Oct 30 12:56:12.641237 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.641115 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea775e41-0676-42f8-937c-5b771342e724-tmp\") pod \"kube-auth-proxy-65dc4d65b9-cvtrj\" (UID: \"ea775e41-0676-42f8-937c-5b771342e724\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-cvtrj" Oct 30 12:56:12.742615 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.742533 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9qpt\" (UniqueName: \"kubernetes.io/projected/ea775e41-0676-42f8-937c-5b771342e724-kube-api-access-l9qpt\") pod \"kube-auth-proxy-65dc4d65b9-cvtrj\" (UID: \"ea775e41-0676-42f8-937c-5b771342e724\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-cvtrj" Oct 30 12:56:12.742615 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.742583 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea775e41-0676-42f8-937c-5b771342e724-tls-certs\") pod \"kube-auth-proxy-65dc4d65b9-cvtrj\" (UID: \"ea775e41-0676-42f8-937c-5b771342e724\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-cvtrj" Oct 30 12:56:12.742778 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.742636 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea775e41-0676-42f8-937c-5b771342e724-tmp\") pod \"kube-auth-proxy-65dc4d65b9-cvtrj\" (UID: \"ea775e41-0676-42f8-937c-5b771342e724\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-cvtrj" Oct 30 12:56:12.744813 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.744792 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea775e41-0676-42f8-937c-5b771342e724-tmp\") pod \"kube-auth-proxy-65dc4d65b9-cvtrj\" (UID: \"ea775e41-0676-42f8-937c-5b771342e724\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-cvtrj" Oct 30 12:56:12.745107 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.745086 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea775e41-0676-42f8-937c-5b771342e724-tls-certs\") pod \"kube-auth-proxy-65dc4d65b9-cvtrj\" (UID: \"ea775e41-0676-42f8-937c-5b771342e724\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-cvtrj" Oct 30 12:56:12.766312 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.766288 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9qpt\" (UniqueName: \"kubernetes.io/projected/ea775e41-0676-42f8-937c-5b771342e724-kube-api-access-l9qpt\") pod \"kube-auth-proxy-65dc4d65b9-cvtrj\" (UID: \"ea775e41-0676-42f8-937c-5b771342e724\") " pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-cvtrj" Oct 30 12:56:12.806755 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:12.806732 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-cvtrj" Oct 30 12:56:13.124893 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:13.124860 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65dc4d65b9-cvtrj"] Oct 30 12:56:13.127752 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:56:13.127720 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea775e41_0676_42f8_937c_5b771342e724.slice/crio-3f280e81b9f64ccb0c1459cdabad221833a00fcef6bfb3e70bfdbd28cf822f47 WatchSource:0}: Error finding container 3f280e81b9f64ccb0c1459cdabad221833a00fcef6bfb3e70bfdbd28cf822f47: Status 404 returned error can't find the container with id 3f280e81b9f64ccb0c1459cdabad221833a00fcef6bfb3e70bfdbd28cf822f47 Oct 30 12:56:13.418877 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:13.418785 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-cvtrj" event={"ID":"ea775e41-0676-42f8-937c-5b771342e724","Type":"ContainerStarted","Data":"3f280e81b9f64ccb0c1459cdabad221833a00fcef6bfb3e70bfdbd28cf822f47"} Oct 30 12:56:14.635730 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.635694 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq"] Oct 30 12:56:14.641914 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.641891 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" Oct 30 12:56:14.662958 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.662931 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq"] Oct 30 12:56:14.698771 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.698737 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 12:56:14.701337 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.701314 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 12:56:14.701472 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.701416 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rwc5b\"" Oct 30 12:56:14.701564 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.701547 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 12:56:14.765563 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.765536 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c95b4635-23a5-401a-b6f8-2b466aa3e3e6-manager-config\") pod \"lws-controller-manager-6546cc79dc-gxqfq\" (UID: \"c95b4635-23a5-401a-b6f8-2b466aa3e3e6\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" Oct 30 12:56:14.765710 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.765585 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c95b4635-23a5-401a-b6f8-2b466aa3e3e6-cert\") pod \"lws-controller-manager-6546cc79dc-gxqfq\" (UID: \"c95b4635-23a5-401a-b6f8-2b466aa3e3e6\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" Oct 30 12:56:14.765710 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.765605 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bwpk\" (UniqueName: \"kubernetes.io/projected/c95b4635-23a5-401a-b6f8-2b466aa3e3e6-kube-api-access-7bwpk\") pod \"lws-controller-manager-6546cc79dc-gxqfq\" (UID: \"c95b4635-23a5-401a-b6f8-2b466aa3e3e6\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" Oct 30 12:56:14.765801 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.765742 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c95b4635-23a5-401a-b6f8-2b466aa3e3e6-metrics-cert\") pod \"lws-controller-manager-6546cc79dc-gxqfq\" (UID: \"c95b4635-23a5-401a-b6f8-2b466aa3e3e6\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" Oct 30 12:56:14.866579 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.866542 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c95b4635-23a5-401a-b6f8-2b466aa3e3e6-manager-config\") pod \"lws-controller-manager-6546cc79dc-gxqfq\" (UID: \"c95b4635-23a5-401a-b6f8-2b466aa3e3e6\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" Oct 30 12:56:14.866761 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.866605 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c95b4635-23a5-401a-b6f8-2b466aa3e3e6-cert\") pod \"lws-controller-manager-6546cc79dc-gxqfq\" (UID: \"c95b4635-23a5-401a-b6f8-2b466aa3e3e6\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" Oct 30 12:56:14.866761 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.866629 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bwpk\" (UniqueName: \"kubernetes.io/projected/c95b4635-23a5-401a-b6f8-2b466aa3e3e6-kube-api-access-7bwpk\") pod \"lws-controller-manager-6546cc79dc-gxqfq\" (UID: \"c95b4635-23a5-401a-b6f8-2b466aa3e3e6\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" Oct 30 12:56:14.866761 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.866666 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c95b4635-23a5-401a-b6f8-2b466aa3e3e6-metrics-cert\") pod \"lws-controller-manager-6546cc79dc-gxqfq\" (UID: \"c95b4635-23a5-401a-b6f8-2b466aa3e3e6\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" Oct 30 12:56:14.867257 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.867227 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c95b4635-23a5-401a-b6f8-2b466aa3e3e6-manager-config\") pod \"lws-controller-manager-6546cc79dc-gxqfq\" (UID: \"c95b4635-23a5-401a-b6f8-2b466aa3e3e6\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" Oct 30 12:56:14.868998 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.868974 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c95b4635-23a5-401a-b6f8-2b466aa3e3e6-metrics-cert\") pod \"lws-controller-manager-6546cc79dc-gxqfq\" (UID: \"c95b4635-23a5-401a-b6f8-2b466aa3e3e6\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" Oct 30 12:56:14.869105 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.869046 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c95b4635-23a5-401a-b6f8-2b466aa3e3e6-cert\") pod \"lws-controller-manager-6546cc79dc-gxqfq\" (UID: \"c95b4635-23a5-401a-b6f8-2b466aa3e3e6\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" Oct 30 12:56:14.890964 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.890917 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bwpk\" (UniqueName: \"kubernetes.io/projected/c95b4635-23a5-401a-b6f8-2b466aa3e3e6-kube-api-access-7bwpk\") pod \"lws-controller-manager-6546cc79dc-gxqfq\" (UID: \"c95b4635-23a5-401a-b6f8-2b466aa3e3e6\") " pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" Oct 30 12:56:14.954129 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:14.954101 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" Oct 30 12:56:15.257923 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:15.257849 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq"] Oct 30 12:56:15.271082 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:56:15.271052 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95b4635_23a5_401a_b6f8_2b466aa3e3e6.slice/crio-5f4160a6c46b2a2588a1d40fbb1bf72197df16bd792ddc81273bfa5ed3fb8cb8 WatchSource:0}: Error finding container 5f4160a6c46b2a2588a1d40fbb1bf72197df16bd792ddc81273bfa5ed3fb8cb8: Status 404 returned error can't find the container with id 5f4160a6c46b2a2588a1d40fbb1bf72197df16bd792ddc81273bfa5ed3fb8cb8 Oct 30 12:56:15.429855 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:15.429818 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" event={"ID":"c95b4635-23a5-401a-b6f8-2b466aa3e3e6","Type":"ContainerStarted","Data":"5f4160a6c46b2a2588a1d40fbb1bf72197df16bd792ddc81273bfa5ed3fb8cb8"} Oct 30 12:56:15.431288 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:15.431260 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-cvtrj" event={"ID":"ea775e41-0676-42f8-937c-5b771342e724","Type":"ContainerStarted","Data":"2f6cc1269c23e5470478e79d57114559662041aaf41c8646704b9513c1f3798f"} Oct 30 12:56:15.478018 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:15.477966 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65dc4d65b9-cvtrj" podStartSLOduration=1.326783521 podStartE2EDuration="3.477954225s" totalImagesPullingTime="2.151170704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:15.476679597 +0000 UTC m=+232.108866759" watchObservedRunningTime="2026-10-30 12:56:15.477954225 +0000 UTC m=+232.110141364" Oct 30 12:56:17.442964 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:17.442929 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" event={"ID":"c95b4635-23a5-401a-b6f8-2b466aa3e3e6","Type":"ContainerStarted","Data":"767b555db9622163497bd3a6d21d14a627c4cb79fa6f40bc5c464e8a36876a99"} Oct 30 12:56:17.443407 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:17.443157 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" Oct 30 12:56:17.486699 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:17.486638 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" podStartSLOduration=2.272642075 podStartE2EDuration="3.486623952s" totalImagesPullingTime="1.213981877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:17.485814322 +0000 UTC m=+234.118001484" watchObservedRunningTime="2026-10-30 12:56:17.486623952 +0000 UTC m=+234.118811108" Oct 30 12:56:28.450017 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:56:28.449989 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6546cc79dc-gxqfq" Oct 30 12:56:29.397381 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:56:29.397341 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 12:56:59.397186 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:56:59.397148 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 12:57:15.746046 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:15.746006 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz"] Oct 30 12:57:15.748920 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:15.748903 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" Oct 30 12:57:15.777806 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:15.777776 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea52cc4c-d370-4d78-b3ee-6ea905918ca2-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-vmtjz\" (UID: \"ea52cc4c-d370-4d78-b3ee-6ea905918ca2\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" Oct 30 12:57:15.777975 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:15.777831 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5flhl\" (UniqueName: \"kubernetes.io/projected/ea52cc4c-d370-4d78-b3ee-6ea905918ca2-kube-api-access-5flhl\") pod \"llmisvc-controller-manager-869d5dcc5d-vmtjz\" (UID: \"ea52cc4c-d370-4d78-b3ee-6ea905918ca2\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" Oct 30 12:57:15.788132 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:15.788102 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-dpzv4\"" Oct 30 12:57:15.788314 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:15.788185 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 12:57:15.790667 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:15.790640 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz"] Oct 30 12:57:15.878287 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:15.878251 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea52cc4c-d370-4d78-b3ee-6ea905918ca2-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-vmtjz\" (UID: \"ea52cc4c-d370-4d78-b3ee-6ea905918ca2\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" Oct 30 12:57:15.878473 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:15.878324 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5flhl\" (UniqueName: \"kubernetes.io/projected/ea52cc4c-d370-4d78-b3ee-6ea905918ca2-kube-api-access-5flhl\") pod \"llmisvc-controller-manager-869d5dcc5d-vmtjz\" (UID: \"ea52cc4c-d370-4d78-b3ee-6ea905918ca2\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" Oct 30 12:57:15.880961 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:15.880934 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea52cc4c-d370-4d78-b3ee-6ea905918ca2-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-vmtjz\" (UID: \"ea52cc4c-d370-4d78-b3ee-6ea905918ca2\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" Oct 30 12:57:15.902693 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:15.902653 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5flhl\" (UniqueName: \"kubernetes.io/projected/ea52cc4c-d370-4d78-b3ee-6ea905918ca2-kube-api-access-5flhl\") pod \"llmisvc-controller-manager-869d5dcc5d-vmtjz\" (UID: \"ea52cc4c-d370-4d78-b3ee-6ea905918ca2\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" Oct 30 12:57:16.059673 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:16.059573 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" Oct 30 12:57:16.373331 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:16.373300 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz"] Oct 30 12:57:16.376386 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:57:16.376353 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podea52cc4c_d370_4d78_b3ee_6ea905918ca2.slice/crio-dde09f3f4f1c90535f81946f05ead443aab11b89723fc4865f52dfd6813202bb WatchSource:0}: Error finding container dde09f3f4f1c90535f81946f05ead443aab11b89723fc4865f52dfd6813202bb: Status 404 returned error can't find the container with id dde09f3f4f1c90535f81946f05ead443aab11b89723fc4865f52dfd6813202bb Oct 30 12:57:16.676998 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:16.676908 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" event={"ID":"ea52cc4c-d370-4d78-b3ee-6ea905918ca2","Type":"ContainerStarted","Data":"dde09f3f4f1c90535f81946f05ead443aab11b89723fc4865f52dfd6813202bb"} Oct 30 12:57:18.687182 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:18.687150 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" event={"ID":"ea52cc4c-d370-4d78-b3ee-6ea905918ca2","Type":"ContainerStarted","Data":"ebfd52241ae720f0c8cd9bf7a7e9792a8fb0859a42da95706dd22187d954f8cb"} Oct 30 12:57:18.687695 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:18.687287 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" Oct 30 12:57:18.791086 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:18.791024 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" podStartSLOduration=2.061746091 podStartE2EDuration="3.791010658s" totalImagesPullingTime="1.729264567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:18.790079357 +0000 UTC m=+295.422266519" watchObservedRunningTime="2026-10-30 12:57:18.791010658 +0000 UTC m=+295.423197821" Oct 30 12:57:23.519308 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:23.519283 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 12:57:23.520987 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:23.520958 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 12:57:29.397586 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:57:29.397549 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 12:57:49.694178 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:57:49.694142 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" Oct 30 12:57:59.398111 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:57:59.398068 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 12:58:29.396756 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:58:29.396710 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 12:58:34.926797 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:34.926714 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rzvww"] Oct 30 12:58:34.930159 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:34.930139 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rzvww" Oct 30 12:58:34.966262 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:34.966240 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 12:58:34.966374 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:34.966357 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 12:58:34.970923 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:34.970908 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 12:58:34.976990 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:34.976977 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hj987\"" Oct 30 12:58:34.977044 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:34.977038 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 12:58:34.997233 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:34.997200 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rzvww"] Oct 30 12:58:35.038869 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:35.038838 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4487\" (UniqueName: \"kubernetes.io/projected/7931e956-afe1-40cb-93b2-d06d02c0b35a-kube-api-access-c4487\") pod \"kuadrant-console-plugin-66c4fb575d-rzvww\" (UID: \"7931e956-afe1-40cb-93b2-d06d02c0b35a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rzvww" Oct 30 12:58:35.038992 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:35.038886 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7931e956-afe1-40cb-93b2-d06d02c0b35a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rzvww\" (UID: \"7931e956-afe1-40cb-93b2-d06d02c0b35a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rzvww" Oct 30 12:58:35.039045 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:35.039022 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7931e956-afe1-40cb-93b2-d06d02c0b35a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rzvww\" (UID: \"7931e956-afe1-40cb-93b2-d06d02c0b35a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rzvww" Oct 30 12:58:35.139952 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:35.139924 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7931e956-afe1-40cb-93b2-d06d02c0b35a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rzvww\" (UID: \"7931e956-afe1-40cb-93b2-d06d02c0b35a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rzvww" Oct 30 12:58:35.140078 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:35.140020 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7931e956-afe1-40cb-93b2-d06d02c0b35a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rzvww\" (UID: \"7931e956-afe1-40cb-93b2-d06d02c0b35a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rzvww" Oct 30 12:58:35.140078 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:35.140066 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4487\" (UniqueName: \"kubernetes.io/projected/7931e956-afe1-40cb-93b2-d06d02c0b35a-kube-api-access-c4487\") pod \"kuadrant-console-plugin-66c4fb575d-rzvww\" (UID: \"7931e956-afe1-40cb-93b2-d06d02c0b35a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rzvww" Oct 30 12:58:35.140619 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:35.140601 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7931e956-afe1-40cb-93b2-d06d02c0b35a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rzvww\" (UID: \"7931e956-afe1-40cb-93b2-d06d02c0b35a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rzvww" Oct 30 12:58:35.142428 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:35.142411 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7931e956-afe1-40cb-93b2-d06d02c0b35a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rzvww\" (UID: \"7931e956-afe1-40cb-93b2-d06d02c0b35a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rzvww" Oct 30 12:58:35.193192 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:35.193140 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4487\" (UniqueName: \"kubernetes.io/projected/7931e956-afe1-40cb-93b2-d06d02c0b35a-kube-api-access-c4487\") pod \"kuadrant-console-plugin-66c4fb575d-rzvww\" (UID: \"7931e956-afe1-40cb-93b2-d06d02c0b35a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rzvww" Oct 30 12:58:35.241059 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:35.241036 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rzvww" Oct 30 12:58:35.549814 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:35.549741 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rzvww"] Oct 30 12:58:35.553782 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:58:35.553756 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7931e956_afe1_40cb_93b2_d06d02c0b35a.slice/crio-ecb279850b2d061c1fed6e7da8bec630796a3e64df9f1c6f15605f84aa62d790 WatchSource:0}: Error finding container ecb279850b2d061c1fed6e7da8bec630796a3e64df9f1c6f15605f84aa62d790: Status 404 returned error can't find the container with id ecb279850b2d061c1fed6e7da8bec630796a3e64df9f1c6f15605f84aa62d790 Oct 30 12:58:36.006974 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:36.006937 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rzvww" event={"ID":"7931e956-afe1-40cb-93b2-d06d02c0b35a","Type":"ContainerStarted","Data":"ecb279850b2d061c1fed6e7da8bec630796a3e64df9f1c6f15605f84aa62d790"} Oct 30 12:58:40.027833 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:40.027795 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rzvww" event={"ID":"7931e956-afe1-40cb-93b2-d06d02c0b35a","Type":"ContainerStarted","Data":"c94fb007e16e56cbfac0cf025d9f02b5e1f579e8832e9d4f3374c5d4f52e69bb"} Oct 30 12:58:40.070391 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:58:40.070341 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rzvww" podStartSLOduration=1.762683779 podStartE2EDuration="6.070329913s" totalImagesPullingTime="4.307646134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:40.069293209 +0000 UTC m=+376.701480371" watchObservedRunningTime="2026-10-30 12:58:40.070329913 +0000 UTC m=+376.702517076" Oct 30 12:58:59.398106 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:58:59.398061 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 12:59:29.397098 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:59:29.397058 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 12:59:31.596117 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:31.596081 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-76tnz"] Oct 30 12:59:31.615604 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:31.615577 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" Oct 30 12:59:31.620756 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:31.620730 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-76tnz"] Oct 30 12:59:31.631684 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:31.631660 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 12:59:31.732084 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:31.732038 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdxrk\" (UniqueName: \"kubernetes.io/projected/2563cc92-295b-4fec-9bb0-dd1a41ee61f9-kube-api-access-bdxrk\") pod \"limitador-limitador-74dfcfb894-76tnz\" (UID: \"2563cc92-295b-4fec-9bb0-dd1a41ee61f9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" Oct 30 12:59:31.732244 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:31.732099 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2563cc92-295b-4fec-9bb0-dd1a41ee61f9-config-file\") pod \"limitador-limitador-74dfcfb894-76tnz\" (UID: \"2563cc92-295b-4fec-9bb0-dd1a41ee61f9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" Oct 30 12:59:31.735945 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:31.735919 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-76tnz"] Oct 30 12:59:31.832913 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:31.832886 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2563cc92-295b-4fec-9bb0-dd1a41ee61f9-config-file\") pod \"limitador-limitador-74dfcfb894-76tnz\" (UID: \"2563cc92-295b-4fec-9bb0-dd1a41ee61f9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" Oct 30 12:59:31.833052 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:31.832966 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdxrk\" (UniqueName: \"kubernetes.io/projected/2563cc92-295b-4fec-9bb0-dd1a41ee61f9-kube-api-access-bdxrk\") pod \"limitador-limitador-74dfcfb894-76tnz\" (UID: \"2563cc92-295b-4fec-9bb0-dd1a41ee61f9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" Oct 30 12:59:31.833583 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:31.833564 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2563cc92-295b-4fec-9bb0-dd1a41ee61f9-config-file\") pod \"limitador-limitador-74dfcfb894-76tnz\" (UID: \"2563cc92-295b-4fec-9bb0-dd1a41ee61f9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" Oct 30 12:59:31.855080 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:31.855017 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdxrk\" (UniqueName: \"kubernetes.io/projected/2563cc92-295b-4fec-9bb0-dd1a41ee61f9-kube-api-access-bdxrk\") pod \"limitador-limitador-74dfcfb894-76tnz\" (UID: \"2563cc92-295b-4fec-9bb0-dd1a41ee61f9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" Oct 30 12:59:31.929651 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:31.929629 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" Oct 30 12:59:32.235129 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:32.235090 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-76tnz"] Oct 30 12:59:32.238404 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:59:32.238372 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2563cc92_295b_4fec_9bb0_dd1a41ee61f9.slice/crio-660c66c9b2f443a07fa61e4ee3e0aa558ec7170576b05b6ca3a3f6be4c868360 WatchSource:0}: Error finding container 660c66c9b2f443a07fa61e4ee3e0aa558ec7170576b05b6ca3a3f6be4c868360: Status 404 returned error can't find the container with id 660c66c9b2f443a07fa61e4ee3e0aa558ec7170576b05b6ca3a3f6be4c868360 Oct 30 12:59:33.248404 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:33.248357 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" event={"ID":"2563cc92-295b-4fec-9bb0-dd1a41ee61f9","Type":"ContainerStarted","Data":"660c66c9b2f443a07fa61e4ee3e0aa558ec7170576b05b6ca3a3f6be4c868360"} Oct 30 12:59:35.257852 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:35.257818 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" event={"ID":"2563cc92-295b-4fec-9bb0-dd1a41ee61f9","Type":"ContainerStarted","Data":"3158165f7d2000c3bff005a0770042455aad77c47edbea2ac7f028620751282f"} Oct 30 12:59:35.258256 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:35.257928 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" Oct 30 12:59:35.296696 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:35.296648 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" podStartSLOduration=2.264675267 podStartE2EDuration="4.296630248s" totalImagesPullingTime="2.031954981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:35.295562317 +0000 UTC m=+431.927749497" watchObservedRunningTime="2026-10-30 12:59:35.296630248 +0000 UTC m=+431.928817406" Oct 30 12:59:46.263566 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:46.263536 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" Oct 30 12:59:47.114926 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:47.114890 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-76tnz"] Oct 30 12:59:47.115129 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:47.115092 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" podUID="2563cc92-295b-4fec-9bb0-dd1a41ee61f9" containerName="limitador" containerID="cri-o://3158165f7d2000c3bff005a0770042455aad77c47edbea2ac7f028620751282f" gracePeriod=30 Oct 30 12:59:47.669665 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:47.669635 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" Oct 30 12:59:47.774054 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:47.773970 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2563cc92-295b-4fec-9bb0-dd1a41ee61f9-config-file\" (UniqueName: \"kubernetes.io/configmap/2563cc92-295b-4fec-9bb0-dd1a41ee61f9-config-file\") pod \"2563cc92-295b-4fec-9bb0-dd1a41ee61f9\" (UID: \"2563cc92-295b-4fec-9bb0-dd1a41ee61f9\") " Oct 30 12:59:47.774054 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:47.774037 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2563cc92-295b-4fec-9bb0-dd1a41ee61f9-kube-api-access-bdxrk\" (UniqueName: \"kubernetes.io/projected/2563cc92-295b-4fec-9bb0-dd1a41ee61f9-kube-api-access-bdxrk\") pod \"2563cc92-295b-4fec-9bb0-dd1a41ee61f9\" (UID: \"2563cc92-295b-4fec-9bb0-dd1a41ee61f9\") " Oct 30 12:59:47.774337 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:47.774314 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2563cc92-295b-4fec-9bb0-dd1a41ee61f9-config-file" pod "2563cc92-295b-4fec-9bb0-dd1a41ee61f9" (UID: "2563cc92-295b-4fec-9bb0-dd1a41ee61f9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:59:47.776161 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:47.776130 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2563cc92-295b-4fec-9bb0-dd1a41ee61f9-kube-api-access-bdxrk" pod "2563cc92-295b-4fec-9bb0-dd1a41ee61f9" (UID: "2563cc92-295b-4fec-9bb0-dd1a41ee61f9"). InnerVolumeSpecName "kube-api-access-bdxrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:59:47.874913 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:47.874876 7010 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2563cc92-295b-4fec-9bb0-dd1a41ee61f9-config-file\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:59:47.874913 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:47.874908 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdxrk\" (UniqueName: \"kubernetes.io/projected/2563cc92-295b-4fec-9bb0-dd1a41ee61f9-kube-api-access-bdxrk\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 12:59:48.313200 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:48.313165 7010 generic.go:356] "Generic (PLEG): container finished" podID="2563cc92-295b-4fec-9bb0-dd1a41ee61f9" containerID="3158165f7d2000c3bff005a0770042455aad77c47edbea2ac7f028620751282f" exitCode=0 Oct 30 12:59:48.313385 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:48.313222 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" Oct 30 12:59:48.313385 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:48.313259 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" event={"ID":"2563cc92-295b-4fec-9bb0-dd1a41ee61f9","Type":"ContainerDied","Data":"3158165f7d2000c3bff005a0770042455aad77c47edbea2ac7f028620751282f"} Oct 30 12:59:48.313385 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:48.313336 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-76tnz" event={"ID":"2563cc92-295b-4fec-9bb0-dd1a41ee61f9","Type":"ContainerDied","Data":"660c66c9b2f443a07fa61e4ee3e0aa558ec7170576b05b6ca3a3f6be4c868360"} Oct 30 12:59:48.313385 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:48.313359 7010 scope.go:122] "RemoveContainer" containerID="3158165f7d2000c3bff005a0770042455aad77c47edbea2ac7f028620751282f" Oct 30 12:59:48.321830 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:48.321812 7010 scope.go:122] "RemoveContainer" containerID="3158165f7d2000c3bff005a0770042455aad77c47edbea2ac7f028620751282f" Oct 30 12:59:48.322136 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:59:48.322116 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3158165f7d2000c3bff005a0770042455aad77c47edbea2ac7f028620751282f\": container with ID starting with 3158165f7d2000c3bff005a0770042455aad77c47edbea2ac7f028620751282f not found: ID does not exist" containerID="3158165f7d2000c3bff005a0770042455aad77c47edbea2ac7f028620751282f" Oct 30 12:59:48.322184 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:48.322147 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3158165f7d2000c3bff005a0770042455aad77c47edbea2ac7f028620751282f"} err="failed to get container status \"3158165f7d2000c3bff005a0770042455aad77c47edbea2ac7f028620751282f\": rpc error: code = NotFound desc = could not find container \"3158165f7d2000c3bff005a0770042455aad77c47edbea2ac7f028620751282f\": container with ID starting with 3158165f7d2000c3bff005a0770042455aad77c47edbea2ac7f028620751282f not found: ID does not exist" Oct 30 12:59:48.358906 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:48.358869 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-76tnz"] Oct 30 12:59:48.369837 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:48.369794 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-76tnz"] Oct 30 12:59:49.486446 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:49.486410 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2563cc92-295b-4fec-9bb0-dd1a41ee61f9" path="/var/lib/kubelet/pods/2563cc92-295b-4fec-9bb0-dd1a41ee61f9/volumes" Oct 30 12:59:52.202758 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.202717 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qr9vn"] Oct 30 12:59:52.203258 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.203226 7010 cpu_manager.go:414] "containerMap: removing container" podUID="2563cc92-295b-4fec-9bb0-dd1a41ee61f9" containerName="limitador" Oct 30 12:59:52.203258 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.203245 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2563cc92-295b-4fec-9bb0-dd1a41ee61f9" containerName="limitador" Oct 30 12:59:52.203390 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.203381 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="2563cc92-295b-4fec-9bb0-dd1a41ee61f9" containerName="limitador" Oct 30 12:59:52.207914 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.207889 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qr9vn" Oct 30 12:59:52.227248 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.227222 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 12:59:52.228024 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.227998 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qr9vn"] Oct 30 12:59:52.233975 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.233948 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:59:52.234099 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.233999 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-n5bzz\"" Oct 30 12:59:52.234162 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.234116 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:59:52.311423 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.311377 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99wrx\" (UniqueName: \"kubernetes.io/projected/4a989e02-7066-4728-ad39-843d52c81703-kube-api-access-99wrx\") pod \"postgres-7cdbf6dc44-qr9vn\" (UID: \"4a989e02-7066-4728-ad39-843d52c81703\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qr9vn" Oct 30 12:59:52.311637 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.311452 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a989e02-7066-4728-ad39-843d52c81703-data\") pod \"postgres-7cdbf6dc44-qr9vn\" (UID: \"4a989e02-7066-4728-ad39-843d52c81703\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qr9vn" Oct 30 12:59:52.412900 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.412857 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99wrx\" (UniqueName: \"kubernetes.io/projected/4a989e02-7066-4728-ad39-843d52c81703-kube-api-access-99wrx\") pod \"postgres-7cdbf6dc44-qr9vn\" (UID: \"4a989e02-7066-4728-ad39-843d52c81703\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qr9vn" Oct 30 12:59:52.413111 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.412919 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a989e02-7066-4728-ad39-843d52c81703-data\") pod \"postgres-7cdbf6dc44-qr9vn\" (UID: \"4a989e02-7066-4728-ad39-843d52c81703\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qr9vn" Oct 30 12:59:52.413351 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.413327 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a989e02-7066-4728-ad39-843d52c81703-data\") pod \"postgres-7cdbf6dc44-qr9vn\" (UID: \"4a989e02-7066-4728-ad39-843d52c81703\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qr9vn" Oct 30 12:59:52.434839 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.434806 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99wrx\" (UniqueName: \"kubernetes.io/projected/4a989e02-7066-4728-ad39-843d52c81703-kube-api-access-99wrx\") pod \"postgres-7cdbf6dc44-qr9vn\" (UID: \"4a989e02-7066-4728-ad39-843d52c81703\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qr9vn" Oct 30 12:59:52.520377 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.520285 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qr9vn" Oct 30 12:59:52.824029 ip-10-0-138-103 kubenswrapper[7010]: W1030 12:59:52.823996 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a989e02_7066_4728_ad39_843d52c81703.slice/crio-b01a8d4050368c2ea27d05ef68e2756ffdfb65eead8ee7bf83f409d7db747e9e WatchSource:0}: Error finding container b01a8d4050368c2ea27d05ef68e2756ffdfb65eead8ee7bf83f409d7db747e9e: Status 404 returned error can't find the container with id b01a8d4050368c2ea27d05ef68e2756ffdfb65eead8ee7bf83f409d7db747e9e Oct 30 12:59:52.827405 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:52.827378 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qr9vn"] Oct 30 12:59:53.343918 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:53.343870 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qr9vn" event={"ID":"4a989e02-7066-4728-ad39-843d52c81703","Type":"ContainerStarted","Data":"b01a8d4050368c2ea27d05ef68e2756ffdfb65eead8ee7bf83f409d7db747e9e"} Oct 30 12:59:58.369584 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:58.369547 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qr9vn" event={"ID":"4a989e02-7066-4728-ad39-843d52c81703","Type":"ContainerStarted","Data":"e85abc8bb2087320fa520aea08a1de1736ee8226fe6455eacc1af95a29d9da76"} Oct 30 12:59:58.370079 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:58.369666 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qr9vn" Oct 30 12:59:58.407179 ip-10-0-138-103 kubenswrapper[7010]: I1030 12:59:58.407120 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qr9vn" podStartSLOduration=1.7804646910000002 podStartE2EDuration="6.407104769s" totalImagesPullingTime="4.626640078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:58.406161014 +0000 UTC m=+455.038348176" watchObservedRunningTime="2026-10-30 12:59:58.407104769 +0000 UTC m=+455.039291931" Oct 30 12:59:59.398931 ip-10-0-138-103 kubenswrapper[7010]: E1030 12:59:59.398888 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:00:04.403058 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:04.402975 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qr9vn" Oct 30 13:00:16.338840 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.338806 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-jvf4b"] Oct 30 13:00:16.355726 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.355701 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-jvf4b" Oct 30 13:00:16.365232 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.365188 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-jvf4b"] Oct 30 13:00:16.374581 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.374546 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 13:00:16.374581 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.374561 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 13:00:16.374581 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.374553 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 13:00:16.374755 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.374637 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h5q88\"" Oct 30 13:00:16.421690 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.421661 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af092bb0-1594-4428-8b7b-be24193a5b9f-webhook-cert\") pod \"maas-controller-577f596567-jvf4b\" (UID: \"af092bb0-1594-4428-8b7b-be24193a5b9f\") " pod="opendatahub/maas-controller-577f596567-jvf4b" Oct 30 13:00:16.421857 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.421734 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq9pc\" (UniqueName: \"kubernetes.io/projected/af092bb0-1594-4428-8b7b-be24193a5b9f-kube-api-access-sq9pc\") pod \"maas-controller-577f596567-jvf4b\" (UID: \"af092bb0-1594-4428-8b7b-be24193a5b9f\") " pod="opendatahub/maas-controller-577f596567-jvf4b" Oct 30 13:00:16.421928 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.421878 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af092bb0-1594-4428-8b7b-be24193a5b9f-metrics-certs\") pod \"maas-controller-577f596567-jvf4b\" (UID: \"af092bb0-1594-4428-8b7b-be24193a5b9f\") " pod="opendatahub/maas-controller-577f596567-jvf4b" Oct 30 13:00:16.523009 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.522976 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af092bb0-1594-4428-8b7b-be24193a5b9f-metrics-certs\") pod \"maas-controller-577f596567-jvf4b\" (UID: \"af092bb0-1594-4428-8b7b-be24193a5b9f\") " pod="opendatahub/maas-controller-577f596567-jvf4b" Oct 30 13:00:16.523009 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.523013 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af092bb0-1594-4428-8b7b-be24193a5b9f-webhook-cert\") pod \"maas-controller-577f596567-jvf4b\" (UID: \"af092bb0-1594-4428-8b7b-be24193a5b9f\") " pod="opendatahub/maas-controller-577f596567-jvf4b" Oct 30 13:00:16.523202 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.523051 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq9pc\" (UniqueName: \"kubernetes.io/projected/af092bb0-1594-4428-8b7b-be24193a5b9f-kube-api-access-sq9pc\") pod \"maas-controller-577f596567-jvf4b\" (UID: \"af092bb0-1594-4428-8b7b-be24193a5b9f\") " pod="opendatahub/maas-controller-577f596567-jvf4b" Oct 30 13:00:16.525400 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.525370 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af092bb0-1594-4428-8b7b-be24193a5b9f-webhook-cert\") pod \"maas-controller-577f596567-jvf4b\" (UID: \"af092bb0-1594-4428-8b7b-be24193a5b9f\") " pod="opendatahub/maas-controller-577f596567-jvf4b" Oct 30 13:00:16.525561 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.525477 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af092bb0-1594-4428-8b7b-be24193a5b9f-metrics-certs\") pod \"maas-controller-577f596567-jvf4b\" (UID: \"af092bb0-1594-4428-8b7b-be24193a5b9f\") " pod="opendatahub/maas-controller-577f596567-jvf4b" Oct 30 13:00:16.545115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.545093 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq9pc\" (UniqueName: \"kubernetes.io/projected/af092bb0-1594-4428-8b7b-be24193a5b9f-kube-api-access-sq9pc\") pod \"maas-controller-577f596567-jvf4b\" (UID: \"af092bb0-1594-4428-8b7b-be24193a5b9f\") " pod="opendatahub/maas-controller-577f596567-jvf4b" Oct 30 13:00:16.668871 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.668846 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-jvf4b" Oct 30 13:00:16.743670 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.743629 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b844f68b7-bf8nh"] Oct 30 13:00:16.747242 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.747215 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b844f68b7-bf8nh" Oct 30 13:00:16.775590 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.775558 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b844f68b7-bf8nh"] Oct 30 13:00:16.825696 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.825663 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b12db1-48f1-4058-94f9-ce7bb7098add-metrics-certs\") pod \"maas-controller-b844f68b7-bf8nh\" (UID: \"67b12db1-48f1-4058-94f9-ce7bb7098add\") " pod="opendatahub/maas-controller-b844f68b7-bf8nh" Oct 30 13:00:16.825696 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.825696 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67b12db1-48f1-4058-94f9-ce7bb7098add-webhook-cert\") pod \"maas-controller-b844f68b7-bf8nh\" (UID: \"67b12db1-48f1-4058-94f9-ce7bb7098add\") " pod="opendatahub/maas-controller-b844f68b7-bf8nh" Oct 30 13:00:16.825919 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.825728 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhzph\" (UniqueName: \"kubernetes.io/projected/67b12db1-48f1-4058-94f9-ce7bb7098add-kube-api-access-rhzph\") pod \"maas-controller-b844f68b7-bf8nh\" (UID: \"67b12db1-48f1-4058-94f9-ce7bb7098add\") " pod="opendatahub/maas-controller-b844f68b7-bf8nh" Oct 30 13:00:16.926579 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.926505 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67b12db1-48f1-4058-94f9-ce7bb7098add-webhook-cert\") pod \"maas-controller-b844f68b7-bf8nh\" (UID: \"67b12db1-48f1-4058-94f9-ce7bb7098add\") " pod="opendatahub/maas-controller-b844f68b7-bf8nh" Oct 30 13:00:16.926579 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.926575 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhzph\" (UniqueName: \"kubernetes.io/projected/67b12db1-48f1-4058-94f9-ce7bb7098add-kube-api-access-rhzph\") pod \"maas-controller-b844f68b7-bf8nh\" (UID: \"67b12db1-48f1-4058-94f9-ce7bb7098add\") " pod="opendatahub/maas-controller-b844f68b7-bf8nh" Oct 30 13:00:16.926764 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.926713 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b12db1-48f1-4058-94f9-ce7bb7098add-metrics-certs\") pod \"maas-controller-b844f68b7-bf8nh\" (UID: \"67b12db1-48f1-4058-94f9-ce7bb7098add\") " pod="opendatahub/maas-controller-b844f68b7-bf8nh" Oct 30 13:00:16.928948 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.928928 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67b12db1-48f1-4058-94f9-ce7bb7098add-webhook-cert\") pod \"maas-controller-b844f68b7-bf8nh\" (UID: \"67b12db1-48f1-4058-94f9-ce7bb7098add\") " pod="opendatahub/maas-controller-b844f68b7-bf8nh" Oct 30 13:00:16.928996 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.928984 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b12db1-48f1-4058-94f9-ce7bb7098add-metrics-certs\") pod \"maas-controller-b844f68b7-bf8nh\" (UID: \"67b12db1-48f1-4058-94f9-ce7bb7098add\") " pod="opendatahub/maas-controller-b844f68b7-bf8nh" Oct 30 13:00:16.949577 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.949556 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhzph\" (UniqueName: \"kubernetes.io/projected/67b12db1-48f1-4058-94f9-ce7bb7098add-kube-api-access-rhzph\") pod \"maas-controller-b844f68b7-bf8nh\" (UID: \"67b12db1-48f1-4058-94f9-ce7bb7098add\") " pod="opendatahub/maas-controller-b844f68b7-bf8nh" Oct 30 13:00:16.968755 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:16.968735 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-jvf4b"] Oct 30 13:00:16.973929 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:00:16.973907 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf092bb0_1594_4428_8b7b_be24193a5b9f.slice/crio-2297e2925d7b8be62c6ccc95cdc82b6216a0a9165471b4d0a02f76a6b349808b WatchSource:0}: Error finding container 2297e2925d7b8be62c6ccc95cdc82b6216a0a9165471b4d0a02f76a6b349808b: Status 404 returned error can't find the container with id 2297e2925d7b8be62c6ccc95cdc82b6216a0a9165471b4d0a02f76a6b349808b Oct 30 13:00:17.071181 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:17.071158 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b844f68b7-bf8nh" Oct 30 13:00:17.376064 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:17.376029 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b844f68b7-bf8nh"] Oct 30 13:00:17.381550 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:00:17.381476 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b12db1_48f1_4058_94f9_ce7bb7098add.slice/crio-c4e342cb4b576554e1f08bf4b83548dde0509655841845a5fe54cf2468e509e1 WatchSource:0}: Error finding container c4e342cb4b576554e1f08bf4b83548dde0509655841845a5fe54cf2468e509e1: Status 404 returned error can't find the container with id c4e342cb4b576554e1f08bf4b83548dde0509655841845a5fe54cf2468e509e1 Oct 30 13:00:17.455679 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:17.455639 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b844f68b7-bf8nh" event={"ID":"67b12db1-48f1-4058-94f9-ce7bb7098add","Type":"ContainerStarted","Data":"c4e342cb4b576554e1f08bf4b83548dde0509655841845a5fe54cf2468e509e1"} Oct 30 13:00:17.456984 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:17.456952 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-jvf4b" event={"ID":"af092bb0-1594-4428-8b7b-be24193a5b9f","Type":"ContainerStarted","Data":"2297e2925d7b8be62c6ccc95cdc82b6216a0a9165471b4d0a02f76a6b349808b"} Oct 30 13:00:20.475670 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:20.475635 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b844f68b7-bf8nh" event={"ID":"67b12db1-48f1-4058-94f9-ce7bb7098add","Type":"ContainerStarted","Data":"b48b4dc68c92bb340db7015e629a69f3a42d8a2a3e7a9b547c22e9f6c39b2f33"} Oct 30 13:00:20.476138 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:20.475730 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b844f68b7-bf8nh" Oct 30 13:00:20.477506 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:20.477466 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-jvf4b" event={"ID":"af092bb0-1594-4428-8b7b-be24193a5b9f","Type":"ContainerStarted","Data":"ac682a846650b1f5206232a7feef0680e047cca9775b3b34e022f3aa972a5d1c"} Oct 30 13:00:20.477627 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:20.477529 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-jvf4b" Oct 30 13:00:20.520463 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:20.520417 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-b844f68b7-bf8nh" podStartSLOduration=1.9523185330000001 podStartE2EDuration="4.520405519s" totalImagesPullingTime="2.568086986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:20.518861593 +0000 UTC m=+477.151048754" watchObservedRunningTime="2026-10-30 13:00:20.520405519 +0000 UTC m=+477.152592680" Oct 30 13:00:29.398720 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:00:29.398680 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:00:31.486233 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:31.486201 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-jvf4b" Oct 30 13:00:31.486608 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:31.486275 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b844f68b7-bf8nh" Oct 30 13:00:31.528109 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:31.528048 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-jvf4b" podStartSLOduration=12.819261181 podStartE2EDuration="15.528031236s" totalImagesPullingTime="2.708770055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:20.569317041 +0000 UTC m=+477.201504205" watchObservedRunningTime="2026-10-30 13:00:31.528031236 +0000 UTC m=+488.160218399" Oct 30 13:00:31.603906 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:31.603874 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-jvf4b"] Oct 30 13:00:31.604199 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:31.604140 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-jvf4b" podUID="af092bb0-1594-4428-8b7b-be24193a5b9f" containerName="manager" containerID="cri-o://ac682a846650b1f5206232a7feef0680e047cca9775b3b34e022f3aa972a5d1c" gracePeriod=30 Oct 30 13:00:31.842812 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:31.842788 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-jvf4b" Oct 30 13:00:31.994799 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:31.994766 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af092bb0-1594-4428-8b7b-be24193a5b9f-kube-api-access-sq9pc\" (UniqueName: \"kubernetes.io/projected/af092bb0-1594-4428-8b7b-be24193a5b9f-kube-api-access-sq9pc\") pod \"af092bb0-1594-4428-8b7b-be24193a5b9f\" (UID: \"af092bb0-1594-4428-8b7b-be24193a5b9f\") " Oct 30 13:00:31.994997 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:31.994828 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af092bb0-1594-4428-8b7b-be24193a5b9f-webhook-cert\" (UniqueName: \"kubernetes.io/secret/af092bb0-1594-4428-8b7b-be24193a5b9f-webhook-cert\") pod \"af092bb0-1594-4428-8b7b-be24193a5b9f\" (UID: \"af092bb0-1594-4428-8b7b-be24193a5b9f\") " Oct 30 13:00:31.994997 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:31.994850 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af092bb0-1594-4428-8b7b-be24193a5b9f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af092bb0-1594-4428-8b7b-be24193a5b9f-metrics-certs\") pod \"af092bb0-1594-4428-8b7b-be24193a5b9f\" (UID: \"af092bb0-1594-4428-8b7b-be24193a5b9f\") " Oct 30 13:00:31.997121 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:31.997079 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af092bb0-1594-4428-8b7b-be24193a5b9f-metrics-certs" pod "af092bb0-1594-4428-8b7b-be24193a5b9f" (UID: "af092bb0-1594-4428-8b7b-be24193a5b9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:00:31.997220 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:31.997095 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af092bb0-1594-4428-8b7b-be24193a5b9f-webhook-cert" pod "af092bb0-1594-4428-8b7b-be24193a5b9f" (UID: "af092bb0-1594-4428-8b7b-be24193a5b9f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:00:31.997220 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:31.997148 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af092bb0-1594-4428-8b7b-be24193a5b9f-kube-api-access-sq9pc" pod "af092bb0-1594-4428-8b7b-be24193a5b9f" (UID: "af092bb0-1594-4428-8b7b-be24193a5b9f"). InnerVolumeSpecName "kube-api-access-sq9pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:32.096518 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:32.096394 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq9pc\" (UniqueName: \"kubernetes.io/projected/af092bb0-1594-4428-8b7b-be24193a5b9f-kube-api-access-sq9pc\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:00:32.096518 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:32.096440 7010 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af092bb0-1594-4428-8b7b-be24193a5b9f-webhook-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:00:32.096518 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:32.096454 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af092bb0-1594-4428-8b7b-be24193a5b9f-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:00:32.532348 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:32.532314 7010 generic.go:356] "Generic (PLEG): container finished" podID="af092bb0-1594-4428-8b7b-be24193a5b9f" containerID="ac682a846650b1f5206232a7feef0680e047cca9775b3b34e022f3aa972a5d1c" exitCode=0 Oct 30 13:00:32.532768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:32.532366 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-jvf4b" Oct 30 13:00:32.532768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:32.532369 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-jvf4b" event={"ID":"af092bb0-1594-4428-8b7b-be24193a5b9f","Type":"ContainerDied","Data":"ac682a846650b1f5206232a7feef0680e047cca9775b3b34e022f3aa972a5d1c"} Oct 30 13:00:32.532768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:32.532518 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-jvf4b" event={"ID":"af092bb0-1594-4428-8b7b-be24193a5b9f","Type":"ContainerDied","Data":"2297e2925d7b8be62c6ccc95cdc82b6216a0a9165471b4d0a02f76a6b349808b"} Oct 30 13:00:32.532768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:32.532556 7010 scope.go:122] "RemoveContainer" containerID="ac682a846650b1f5206232a7feef0680e047cca9775b3b34e022f3aa972a5d1c" Oct 30 13:00:32.541093 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:32.541076 7010 scope.go:122] "RemoveContainer" containerID="ac682a846650b1f5206232a7feef0680e047cca9775b3b34e022f3aa972a5d1c" Oct 30 13:00:32.541379 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:00:32.541360 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac682a846650b1f5206232a7feef0680e047cca9775b3b34e022f3aa972a5d1c\": container with ID starting with ac682a846650b1f5206232a7feef0680e047cca9775b3b34e022f3aa972a5d1c not found: ID does not exist" containerID="ac682a846650b1f5206232a7feef0680e047cca9775b3b34e022f3aa972a5d1c" Oct 30 13:00:32.541426 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:32.541390 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac682a846650b1f5206232a7feef0680e047cca9775b3b34e022f3aa972a5d1c"} err="failed to get container status \"ac682a846650b1f5206232a7feef0680e047cca9775b3b34e022f3aa972a5d1c\": rpc error: code = NotFound desc = could not find container \"ac682a846650b1f5206232a7feef0680e047cca9775b3b34e022f3aa972a5d1c\": container with ID starting with ac682a846650b1f5206232a7feef0680e047cca9775b3b34e022f3aa972a5d1c not found: ID does not exist" Oct 30 13:00:32.578934 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:32.578899 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-jvf4b"] Oct 30 13:00:32.589691 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:32.589662 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-jvf4b"] Oct 30 13:00:33.486203 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:33.486168 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af092bb0-1594-4428-8b7b-be24193a5b9f" path="/var/lib/kubelet/pods/af092bb0-1594-4428-8b7b-be24193a5b9f/volumes" Oct 30 13:00:56.879590 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:56.879557 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v"] Oct 30 13:00:56.880189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:56.880081 7010 cpu_manager.go:414] "containerMap: removing container" podUID="af092bb0-1594-4428-8b7b-be24193a5b9f" containerName="manager" Oct 30 13:00:56.880189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:56.880098 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af092bb0-1594-4428-8b7b-be24193a5b9f" containerName="manager" Oct 30 13:00:56.880189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:56.880185 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="af092bb0-1594-4428-8b7b-be24193a5b9f" containerName="manager" Oct 30 13:00:56.886298 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:56.886275 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:00:56.906265 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:56.906237 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d5d642-8a19-42aa-acdd-61ea5c8960bf-maas-api-tls\") pod \"maas-api-5c48877c75-4zm8v\" (UID: \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:00:56.906399 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:56.906285 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skmwn\" (UniqueName: \"kubernetes.io/projected/64d5d642-8a19-42aa-acdd-61ea5c8960bf-kube-api-access-skmwn\") pod \"maas-api-5c48877c75-4zm8v\" (UID: \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:00:56.906399 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:56.906367 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d5d642-8a19-42aa-acdd-61ea5c8960bf-openshift-service-ca\") pod \"maas-api-5c48877c75-4zm8v\" (UID: \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:00:56.906574 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:56.906468 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d5d642-8a19-42aa-acdd-61ea5c8960bf-metrics-certs\") pod \"maas-api-5c48877c75-4zm8v\" (UID: \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:00:56.912739 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:56.912716 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 13:00:56.912861 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:56.912745 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 13:00:56.912861 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:56.912787 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zw9vx\"" Oct 30 13:00:56.914294 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:56.914274 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v"] Oct 30 13:00:57.007180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:57.007143 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d5d642-8a19-42aa-acdd-61ea5c8960bf-metrics-certs\") pod \"maas-api-5c48877c75-4zm8v\" (UID: \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:00:57.007378 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:57.007208 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d5d642-8a19-42aa-acdd-61ea5c8960bf-maas-api-tls\") pod \"maas-api-5c48877c75-4zm8v\" (UID: \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:00:57.007378 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:57.007246 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skmwn\" (UniqueName: \"kubernetes.io/projected/64d5d642-8a19-42aa-acdd-61ea5c8960bf-kube-api-access-skmwn\") pod \"maas-api-5c48877c75-4zm8v\" (UID: \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:00:57.007378 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:57.007278 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d5d642-8a19-42aa-acdd-61ea5c8960bf-openshift-service-ca\") pod \"maas-api-5c48877c75-4zm8v\" (UID: \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:00:57.007955 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:57.007929 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d5d642-8a19-42aa-acdd-61ea5c8960bf-openshift-service-ca\") pod \"maas-api-5c48877c75-4zm8v\" (UID: \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:00:57.009741 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:57.009713 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d5d642-8a19-42aa-acdd-61ea5c8960bf-maas-api-tls\") pod \"maas-api-5c48877c75-4zm8v\" (UID: \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:00:57.009859 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:57.009829 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d5d642-8a19-42aa-acdd-61ea5c8960bf-metrics-certs\") pod \"maas-api-5c48877c75-4zm8v\" (UID: \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:00:57.029745 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:57.029719 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skmwn\" (UniqueName: \"kubernetes.io/projected/64d5d642-8a19-42aa-acdd-61ea5c8960bf-kube-api-access-skmwn\") pod \"maas-api-5c48877c75-4zm8v\" (UID: \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:00:57.199579 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:57.199482 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:00:57.504714 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:57.504583 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v"] Oct 30 13:00:57.508739 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:00:57.508702 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d5d642_8a19_42aa_acdd_61ea5c8960bf.slice/crio-23a7e2a988a3a73db25b53095eb7ddbb65abd54573064bbd658c3124ed16edee WatchSource:0}: Error finding container 23a7e2a988a3a73db25b53095eb7ddbb65abd54573064bbd658c3124ed16edee: Status 404 returned error can't find the container with id 23a7e2a988a3a73db25b53095eb7ddbb65abd54573064bbd658c3124ed16edee Oct 30 13:00:57.510035 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:57.510018 7010 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:00:57.641697 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:57.641661 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" event={"ID":"64d5d642-8a19-42aa-acdd-61ea5c8960bf","Type":"ContainerStarted","Data":"23a7e2a988a3a73db25b53095eb7ddbb65abd54573064bbd658c3124ed16edee"} Oct 30 13:00:59.399094 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:00:59.399060 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:00:59.652058 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:59.651968 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" event={"ID":"64d5d642-8a19-42aa-acdd-61ea5c8960bf","Type":"ContainerStarted","Data":"bfa922987f4a6d208b32a94fb289464020ae7721e906be1d28c96b5f37c4a1b9"} Oct 30 13:00:59.652219 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:59.652086 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:00:59.699240 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:00:59.699174 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" podStartSLOduration=2.168552068 podStartE2EDuration="3.6991568s" totalImagesPullingTime="1.530604732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:59.697769155 +0000 UTC m=+516.329956329" watchObservedRunningTime="2026-10-30 13:00:59.6991568 +0000 UTC m=+516.331343961" Oct 30 13:01:05.662251 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:05.662217 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:01:06.723271 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:06.723236 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-59xk4"] Oct 30 13:01:06.726642 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:06.726624 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-59xk4" Oct 30 13:01:06.745666 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:06.745639 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-59xk4"] Oct 30 13:01:06.805857 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:06.805822 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c0c7b30-b37a-471f-bab0-fcbfcdefa069-cert\") pod \"llmisvc-controller-manager-6466c554d8-59xk4\" (UID: \"6c0c7b30-b37a-471f-bab0-fcbfcdefa069\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-59xk4" Oct 30 13:01:06.806033 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:06.805874 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbrwh\" (UniqueName: \"kubernetes.io/projected/6c0c7b30-b37a-471f-bab0-fcbfcdefa069-kube-api-access-hbrwh\") pod \"llmisvc-controller-manager-6466c554d8-59xk4\" (UID: \"6c0c7b30-b37a-471f-bab0-fcbfcdefa069\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-59xk4" Oct 30 13:01:06.906758 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:06.906721 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c0c7b30-b37a-471f-bab0-fcbfcdefa069-cert\") pod \"llmisvc-controller-manager-6466c554d8-59xk4\" (UID: \"6c0c7b30-b37a-471f-bab0-fcbfcdefa069\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-59xk4" Oct 30 13:01:06.906942 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:06.906793 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbrwh\" (UniqueName: \"kubernetes.io/projected/6c0c7b30-b37a-471f-bab0-fcbfcdefa069-kube-api-access-hbrwh\") pod \"llmisvc-controller-manager-6466c554d8-59xk4\" (UID: \"6c0c7b30-b37a-471f-bab0-fcbfcdefa069\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-59xk4" Oct 30 13:01:06.909166 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:06.909140 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c0c7b30-b37a-471f-bab0-fcbfcdefa069-cert\") pod \"llmisvc-controller-manager-6466c554d8-59xk4\" (UID: \"6c0c7b30-b37a-471f-bab0-fcbfcdefa069\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-59xk4" Oct 30 13:01:06.929786 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:06.929747 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbrwh\" (UniqueName: \"kubernetes.io/projected/6c0c7b30-b37a-471f-bab0-fcbfcdefa069-kube-api-access-hbrwh\") pod \"llmisvc-controller-manager-6466c554d8-59xk4\" (UID: \"6c0c7b30-b37a-471f-bab0-fcbfcdefa069\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-59xk4" Oct 30 13:01:07.039514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:07.039405 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-59xk4" Oct 30 13:01:07.345678 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:07.345596 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-59xk4"] Oct 30 13:01:07.350282 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:01:07.350248 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0c7b30_b37a_471f_bab0_fcbfcdefa069.slice/crio-27fd13c1e8f7ee42b7385df5563142bbd7e80a02d3d2cc91e36c02a084736dcb WatchSource:0}: Error finding container 27fd13c1e8f7ee42b7385df5563142bbd7e80a02d3d2cc91e36c02a084736dcb: Status 404 returned error can't find the container with id 27fd13c1e8f7ee42b7385df5563142bbd7e80a02d3d2cc91e36c02a084736dcb Oct 30 13:01:07.687864 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:07.687831 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-59xk4" event={"ID":"6c0c7b30-b37a-471f-bab0-fcbfcdefa069","Type":"ContainerStarted","Data":"22cf0aa7d2ae3bbdacb80dd5dc87626e78349ea893e714d9687fea491cfad8e0"} Oct 30 13:01:07.688013 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:07.687883 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-59xk4" event={"ID":"6c0c7b30-b37a-471f-bab0-fcbfcdefa069","Type":"ContainerStarted","Data":"27fd13c1e8f7ee42b7385df5563142bbd7e80a02d3d2cc91e36c02a084736dcb"} Oct 30 13:01:07.688013 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:07.687925 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-59xk4" Oct 30 13:01:07.726737 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:07.726680 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-59xk4" podStartSLOduration=1.450505521 podStartE2EDuration="1.726665117s" totalImagesPullingTime="276.159596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:07.725664554 +0000 UTC m=+524.357851716" watchObservedRunningTime="2026-10-30 13:01:07.726665117 +0000 UTC m=+524.358852309" Oct 30 13:01:29.398165 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:01:29.398118 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:01:38.693502 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:38.693408 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-59xk4" Oct 30 13:01:38.766921 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:38.766887 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz"] Oct 30 13:01:38.767123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:38.767095 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" podUID="ea52cc4c-d370-4d78-b3ee-6ea905918ca2" containerName="manager" containerID="cri-o://ebfd52241ae720f0c8cd9bf7a7e9792a8fb0859a42da95706dd22187d954f8cb" gracePeriod=30 Oct 30 13:01:39.014067 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:39.014042 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" Oct 30 13:01:39.195561 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:39.195482 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea52cc4c-d370-4d78-b3ee-6ea905918ca2-cert\" (UniqueName: \"kubernetes.io/secret/ea52cc4c-d370-4d78-b3ee-6ea905918ca2-cert\") pod \"ea52cc4c-d370-4d78-b3ee-6ea905918ca2\" (UID: \"ea52cc4c-d370-4d78-b3ee-6ea905918ca2\") " Oct 30 13:01:39.195561 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:39.195551 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea52cc4c-d370-4d78-b3ee-6ea905918ca2-kube-api-access-5flhl\" (UniqueName: \"kubernetes.io/projected/ea52cc4c-d370-4d78-b3ee-6ea905918ca2-kube-api-access-5flhl\") pod \"ea52cc4c-d370-4d78-b3ee-6ea905918ca2\" (UID: \"ea52cc4c-d370-4d78-b3ee-6ea905918ca2\") " Oct 30 13:01:39.197617 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:39.197589 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea52cc4c-d370-4d78-b3ee-6ea905918ca2-cert" pod "ea52cc4c-d370-4d78-b3ee-6ea905918ca2" (UID: "ea52cc4c-d370-4d78-b3ee-6ea905918ca2"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:01:39.197720 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:39.197710 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea52cc4c-d370-4d78-b3ee-6ea905918ca2-kube-api-access-5flhl" pod "ea52cc4c-d370-4d78-b3ee-6ea905918ca2" (UID: "ea52cc4c-d370-4d78-b3ee-6ea905918ca2"). InnerVolumeSpecName "kube-api-access-5flhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:01:39.297198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:39.297115 7010 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea52cc4c-d370-4d78-b3ee-6ea905918ca2-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:01:39.297198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:39.297147 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5flhl\" (UniqueName: \"kubernetes.io/projected/ea52cc4c-d370-4d78-b3ee-6ea905918ca2-kube-api-access-5flhl\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:01:39.827527 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:39.827344 7010 generic.go:356] "Generic (PLEG): container finished" podID="ea52cc4c-d370-4d78-b3ee-6ea905918ca2" containerID="ebfd52241ae720f0c8cd9bf7a7e9792a8fb0859a42da95706dd22187d954f8cb" exitCode=0 Oct 30 13:01:39.827527 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:39.827441 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" event={"ID":"ea52cc4c-d370-4d78-b3ee-6ea905918ca2","Type":"ContainerDied","Data":"ebfd52241ae720f0c8cd9bf7a7e9792a8fb0859a42da95706dd22187d954f8cb"} Oct 30 13:01:39.832466 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:39.828165 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" event={"ID":"ea52cc4c-d370-4d78-b3ee-6ea905918ca2","Type":"ContainerDied","Data":"dde09f3f4f1c90535f81946f05ead443aab11b89723fc4865f52dfd6813202bb"} Oct 30 13:01:39.832466 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:39.828231 7010 scope.go:122] "RemoveContainer" containerID="ebfd52241ae720f0c8cd9bf7a7e9792a8fb0859a42da95706dd22187d954f8cb" Oct 30 13:01:39.832466 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:39.828431 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz" Oct 30 13:01:39.841089 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:39.841059 7010 scope.go:122] "RemoveContainer" containerID="ebfd52241ae720f0c8cd9bf7a7e9792a8fb0859a42da95706dd22187d954f8cb" Oct 30 13:01:39.841403 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:01:39.841382 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebfd52241ae720f0c8cd9bf7a7e9792a8fb0859a42da95706dd22187d954f8cb\": container with ID starting with ebfd52241ae720f0c8cd9bf7a7e9792a8fb0859a42da95706dd22187d954f8cb not found: ID does not exist" containerID="ebfd52241ae720f0c8cd9bf7a7e9792a8fb0859a42da95706dd22187d954f8cb" Oct 30 13:01:39.841454 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:39.841413 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfd52241ae720f0c8cd9bf7a7e9792a8fb0859a42da95706dd22187d954f8cb"} err="failed to get container status \"ebfd52241ae720f0c8cd9bf7a7e9792a8fb0859a42da95706dd22187d954f8cb\": rpc error: code = NotFound desc = could not find container \"ebfd52241ae720f0c8cd9bf7a7e9792a8fb0859a42da95706dd22187d954f8cb\": container with ID starting with ebfd52241ae720f0c8cd9bf7a7e9792a8fb0859a42da95706dd22187d954f8cb not found: ID does not exist" Oct 30 13:01:39.885641 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:39.885609 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz"] Oct 30 13:01:39.896615 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:39.896588 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vmtjz"] Oct 30 13:01:41.486591 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:01:41.486556 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea52cc4c-d370-4d78-b3ee-6ea905918ca2" path="/var/lib/kubelet/pods/ea52cc4c-d370-4d78-b3ee-6ea905918ca2/volumes" Oct 30 13:01:59.398072 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:01:59.398031 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:02:18.173268 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.173229 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585"] Oct 30 13:02:18.173825 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.173806 7010 cpu_manager.go:414] "containerMap: removing container" podUID="ea52cc4c-d370-4d78-b3ee-6ea905918ca2" containerName="manager" Oct 30 13:02:18.173874 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.173827 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea52cc4c-d370-4d78-b3ee-6ea905918ca2" containerName="manager" Oct 30 13:02:18.173915 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.173905 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea52cc4c-d370-4d78-b3ee-6ea905918ca2" containerName="manager" Oct 30 13:02:18.177082 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.177056 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.211088 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.211052 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585"] Oct 30 13:02:18.260827 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.260786 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.260827 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.260832 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.261074 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.260917 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrfdn\" (UniqueName: \"kubernetes.io/projected/db8b7959-d693-43bb-b380-1b1ae29776f2-kube-api-access-lrfdn\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.261074 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.261007 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.261074 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.261039 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.261217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.261088 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.261217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.261114 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8b7959-d693-43bb-b380-1b1ae29776f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.261217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.261186 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.261364 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.261233 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.268801 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.268775 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 13:02:18.268936 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.268872 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bwksz\"" Oct 30 13:02:18.362524 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.362470 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.362704 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.362541 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8b7959-d693-43bb-b380-1b1ae29776f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.362704 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.362580 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.362815 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.362738 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.362875 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.362817 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.362875 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.362858 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.362978 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.362911 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrfdn\" (UniqueName: \"kubernetes.io/projected/db8b7959-d693-43bb-b380-1b1ae29776f2-kube-api-access-lrfdn\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.362978 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.362932 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.362978 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.362952 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.363118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.363010 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.363118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.363038 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.363246 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.363221 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.363347 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.363329 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.363404 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.363378 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8b7959-d693-43bb-b380-1b1ae29776f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.365154 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.365131 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.365352 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.365334 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.391365 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.391332 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.400380 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.400350 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrfdn\" (UniqueName: \"kubernetes.io/projected/db8b7959-d693-43bb-b380-1b1ae29776f2-kube-api-access-lrfdn\") pod \"maas-default-gateway-openshift-default-64d4cc998c-jp585\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.490144 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.490062 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:18.810895 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.810805 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585"] Oct 30 13:02:18.814989 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:02:18.814953 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8b7959_d693_43bb_b380_1b1ae29776f2.slice/crio-22d90d7dae28f61ce65bf4ef7598dd457acf8752058860942b369038aca87910 WatchSource:0}: Error finding container 22d90d7dae28f61ce65bf4ef7598dd457acf8752058860942b369038aca87910: Status 404 returned error can't find the container with id 22d90d7dae28f61ce65bf4ef7598dd457acf8752058860942b369038aca87910 Oct 30 13:02:18.997603 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:18.997568 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" event={"ID":"db8b7959-d693-43bb-b380-1b1ae29776f2","Type":"ContainerStarted","Data":"22d90d7dae28f61ce65bf4ef7598dd457acf8752058860942b369038aca87910"} Oct 30 13:02:22.231783 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:22.231753 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:22.232049 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:22.231823 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:22.232049 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:22.231853 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:02:23.019102 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:23.019064 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" event={"ID":"db8b7959-d693-43bb-b380-1b1ae29776f2","Type":"ContainerStarted","Data":"67aeff2e357d26325df5cc3f35f9dcd58b7fedff85f962bf7d00fd6813eb3c49"} Oct 30 13:02:23.083052 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:23.082991 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" podStartSLOduration=1.6690180959999998 podStartE2EDuration="5.08297493s" totalImagesPullingTime="3.413956834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:23.081550797 +0000 UTC m=+599.713737956" watchObservedRunningTime="2026-10-30 13:02:23.08297493 +0000 UTC m=+599.715162091" Oct 30 13:02:23.498899 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:23.498306 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jp585 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 13:02:23.498899 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:23.498385 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" podUID="db8b7959-d693-43bb-b380-1b1ae29776f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 13:02:23.511803 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:23.511771 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:23.552368 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:23.552343 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:02:23.556781 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:23.556761 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:02:24.505512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:24.498876 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jp585 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 13:02:24.505512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:24.498953 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" podUID="db8b7959-d693-43bb-b380-1b1ae29776f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 13:02:25.505511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:25.497235 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jp585 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 13:02:25.505511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:25.497302 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" podUID="db8b7959-d693-43bb-b380-1b1ae29776f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 13:02:26.507512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:26.496845 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jp585 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 13:02:26.507512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:26.496905 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" podUID="db8b7959-d693-43bb-b380-1b1ae29776f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 13:02:27.495922 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:27.495884 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jp585 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 13:02:27.512516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:27.502660 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" podUID="db8b7959-d693-43bb-b380-1b1ae29776f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 13:02:28.491347 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:28.491307 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jp585 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 13:02:28.491526 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:28.491384 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" podUID="db8b7959-d693-43bb-b380-1b1ae29776f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 13:02:29.401402 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:02:29.401363 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:02:29.491248 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:29.491214 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-jp585 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 13:02:29.491467 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:29.491282 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" podUID="db8b7959-d693-43bb-b380-1b1ae29776f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 13:02:30.558407 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:30.558369 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:30.558826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:30.558793 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:30.559468 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:02:30.559450 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:02:59.399679 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:02:59.399637 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:03:27.487639 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:27.487609 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn"] Oct 30 13:03:27.491350 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:27.491331 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:03:27.511310 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:27.511279 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-52a649-serving-cert\"" Oct 30 13:03:27.523213 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:27.523177 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn"] Oct 30 13:03:27.604806 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:27.604771 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-maas-api-tls\") pod \"maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn\" (UID: \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:03:27.604967 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:27.604820 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66rqp\" (UniqueName: \"kubernetes.io/projected/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-kube-api-access-66rqp\") pod \"maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn\" (UID: \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:03:27.604967 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:27.604939 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn\" (UID: \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:03:27.605051 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:27.604969 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-metrics-certs\") pod \"maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn\" (UID: \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:03:27.705800 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:27.705759 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66rqp\" (UniqueName: \"kubernetes.io/projected/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-kube-api-access-66rqp\") pod \"maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn\" (UID: \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:03:27.706007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:27.705826 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn\" (UID: \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:03:27.706007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:27.705860 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-metrics-certs\") pod \"maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn\" (UID: \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:03:27.706007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:27.705895 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-maas-api-tls\") pod \"maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn\" (UID: \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:03:27.706688 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:27.706626 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn\" (UID: \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:03:27.708859 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:27.708836 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-metrics-certs\") pod \"maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn\" (UID: \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:03:27.708981 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:27.708893 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-maas-api-tls\") pod \"maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn\" (UID: \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:03:27.734954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:27.734926 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66rqp\" (UniqueName: \"kubernetes.io/projected/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-kube-api-access-66rqp\") pod \"maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn\" (UID: \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:03:27.804958 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:27.804872 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:03:28.110847 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:28.110757 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn"] Oct 30 13:03:28.115093 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:03:28.115065 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1350a65_bf30_427d_be17_0ad9bcf8aa0a.slice/crio-0c0fcea8bf29a7614d0b5788468c3054ba4f0aa7cdd3725dca605365dd4c3558 WatchSource:0}: Error finding container 0c0fcea8bf29a7614d0b5788468c3054ba4f0aa7cdd3725dca605365dd4c3558: Status 404 returned error can't find the container with id 0c0fcea8bf29a7614d0b5788468c3054ba4f0aa7cdd3725dca605365dd4c3558 Oct 30 13:03:28.309526 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:28.309461 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" event={"ID":"d1350a65-bf30-427d-be17-0ad9bcf8aa0a","Type":"ContainerStarted","Data":"0c0fcea8bf29a7614d0b5788468c3054ba4f0aa7cdd3725dca605365dd4c3558"} Oct 30 13:03:29.315945 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:29.315910 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" event={"ID":"d1350a65-bf30-427d-be17-0ad9bcf8aa0a","Type":"ContainerStarted","Data":"79745f9109309b212457cdde1bd8b1755584e723aca85380993a494845c9e19e"} Oct 30 13:03:29.316334 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:29.315978 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:03:29.358677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:29.358610 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" podStartSLOduration=2.049364284 podStartE2EDuration="2.358592367s" totalImagesPullingTime="309.228083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:29.35689445 +0000 UTC m=+665.989081633" watchObservedRunningTime="2026-10-30 13:03:29.358592367 +0000 UTC m=+665.990779531" Oct 30 13:03:29.399123 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:03:29.399066 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:03:32.728301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:32.728264 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf"] Oct 30 13:03:32.732084 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:32.732063 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:03:32.762678 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:32.762649 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-ee7d65-serving-cert\"" Oct 30 13:03:32.764678 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:32.764642 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf"] Oct 30 13:03:32.857393 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:32.857354 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-maas-api-tls\") pod \"maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf\" (UID: \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:03:32.857393 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:32.857394 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf\" (UID: \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:03:32.857655 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:32.857545 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdggr\" (UniqueName: \"kubernetes.io/projected/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-kube-api-access-wdggr\") pod \"maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf\" (UID: \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:03:32.857655 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:32.857595 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-metrics-certs\") pod \"maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf\" (UID: \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:03:32.958213 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:32.958178 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-metrics-certs\") pod \"maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf\" (UID: \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:03:32.958427 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:32.958291 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-maas-api-tls\") pod \"maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf\" (UID: \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:03:32.958427 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:32.958323 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf\" (UID: \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:03:32.958427 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:32.958381 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdggr\" (UniqueName: \"kubernetes.io/projected/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-kube-api-access-wdggr\") pod \"maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf\" (UID: \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:03:32.959213 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:32.959177 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf\" (UID: \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:03:32.960998 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:32.960966 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-metrics-certs\") pod \"maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf\" (UID: \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:03:32.961483 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:32.961465 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-maas-api-tls\") pod \"maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf\" (UID: \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:03:32.981287 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:32.981212 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdggr\" (UniqueName: \"kubernetes.io/projected/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-kube-api-access-wdggr\") pod \"maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf\" (UID: \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:03:33.046070 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:33.046033 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:03:33.370734 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:33.370703 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf"] Oct 30 13:03:33.375598 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:03:33.375569 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c6fc6e_3eea_4135_85e2_e68b79d9b5c8.slice/crio-937f2a8fb52696d059be44b29bc83359a7e98fe8f59ad4206c878d6ad596e980 WatchSource:0}: Error finding container 937f2a8fb52696d059be44b29bc83359a7e98fe8f59ad4206c878d6ad596e980: Status 404 returned error can't find the container with id 937f2a8fb52696d059be44b29bc83359a7e98fe8f59ad4206c878d6ad596e980 Oct 30 13:03:34.341127 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:34.341088 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" event={"ID":"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8","Type":"ContainerStarted","Data":"43b92a7406a2ca3115664a6077cfe3c66cc94979dbe6d75b1a8ea9431f2ce4ea"} Oct 30 13:03:34.341571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:34.341140 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" event={"ID":"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8","Type":"ContainerStarted","Data":"937f2a8fb52696d059be44b29bc83359a7e98fe8f59ad4206c878d6ad596e980"} Oct 30 13:03:34.341571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:34.341179 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:03:34.381737 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:34.381685 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" podStartSLOduration=1.955915571 podStartE2EDuration="2.381669173s" totalImagesPullingTime="425.753602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:34.381007426 +0000 UTC m=+671.013194589" watchObservedRunningTime="2026-10-30 13:03:34.381669173 +0000 UTC m=+671.013856337" Oct 30 13:03:34.820846 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:34.820810 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg"] Oct 30 13:03:34.824820 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:34.824799 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:03:34.838361 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:34.838337 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-f07dfe-serving-cert\"" Oct 30 13:03:34.849475 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:34.849443 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg"] Oct 30 13:03:34.979838 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:34.979800 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37fbf9d2-f376-4f7d-a514-54e51342f2f6-metrics-certs\") pod \"maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg\" (UID: \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:03:34.979838 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:34.979838 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37fbf9d2-f376-4f7d-a514-54e51342f2f6-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg\" (UID: \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:03:34.980078 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:34.979954 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vt9s\" (UniqueName: \"kubernetes.io/projected/37fbf9d2-f376-4f7d-a514-54e51342f2f6-kube-api-access-8vt9s\") pod \"maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg\" (UID: \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:03:34.980078 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:34.979994 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37fbf9d2-f376-4f7d-a514-54e51342f2f6-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg\" (UID: \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:03:35.081467 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:35.081382 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37fbf9d2-f376-4f7d-a514-54e51342f2f6-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg\" (UID: \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:03:35.081467 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:35.081451 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37fbf9d2-f376-4f7d-a514-54e51342f2f6-metrics-certs\") pod \"maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg\" (UID: \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:03:35.081737 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:35.081475 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37fbf9d2-f376-4f7d-a514-54e51342f2f6-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg\" (UID: \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:03:35.081737 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:35.081553 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vt9s\" (UniqueName: \"kubernetes.io/projected/37fbf9d2-f376-4f7d-a514-54e51342f2f6-kube-api-access-8vt9s\") pod \"maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg\" (UID: \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:03:35.082202 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:35.082179 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37fbf9d2-f376-4f7d-a514-54e51342f2f6-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg\" (UID: \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:03:35.083947 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:35.083925 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37fbf9d2-f376-4f7d-a514-54e51342f2f6-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg\" (UID: \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:03:35.084036 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:35.083981 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37fbf9d2-f376-4f7d-a514-54e51342f2f6-metrics-certs\") pod \"maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg\" (UID: \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:03:35.103921 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:35.103897 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vt9s\" (UniqueName: \"kubernetes.io/projected/37fbf9d2-f376-4f7d-a514-54e51342f2f6-kube-api-access-8vt9s\") pod \"maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg\" (UID: \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:03:35.137731 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:35.137701 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:03:35.326540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:35.326483 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:03:35.439803 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:35.439769 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg"] Oct 30 13:03:35.444183 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:03:35.444156 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fbf9d2_f376_4f7d_a514_54e51342f2f6.slice/crio-4020d227e099815e1908bef9efcd9efa9e18deb54ffda7923d6f2f700275f6ac WatchSource:0}: Error finding container 4020d227e099815e1908bef9efcd9efa9e18deb54ffda7923d6f2f700275f6ac: Status 404 returned error can't find the container with id 4020d227e099815e1908bef9efcd9efa9e18deb54ffda7923d6f2f700275f6ac Oct 30 13:03:36.353775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:36.353713 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" event={"ID":"37fbf9d2-f376-4f7d-a514-54e51342f2f6","Type":"ContainerStarted","Data":"211ef12859be642ed0c067b0ff4e1df7759429b9ac9c65b0f23a3190186fe54d"} Oct 30 13:03:36.354014 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:36.353836 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" event={"ID":"37fbf9d2-f376-4f7d-a514-54e51342f2f6","Type":"ContainerStarted","Data":"4020d227e099815e1908bef9efcd9efa9e18deb54ffda7923d6f2f700275f6ac"} Oct 30 13:03:36.354200 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:36.354176 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:03:36.394806 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:36.394557 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" podStartSLOduration=2.069409451 podStartE2EDuration="2.394538485s" totalImagesPullingTime="325.129034ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:36.393945137 +0000 UTC m=+673.026132304" watchObservedRunningTime="2026-10-30 13:03:36.394538485 +0000 UTC m=+673.026725648" Oct 30 13:03:40.351060 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:40.351030 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:03:40.953223 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:40.953187 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7"] Oct 30 13:03:40.957044 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:40.957022 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:03:40.972912 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:40.972885 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-cb58b8-serving-cert\"" Oct 30 13:03:40.991866 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:40.991829 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7"] Oct 30 13:03:41.142282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:41.142242 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4z5h\" (UniqueName: \"kubernetes.io/projected/7460fa64-f0e9-4542-a6cd-31d353166cc1-kube-api-access-w4z5h\") pod \"maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7\" (UID: \"7460fa64-f0e9-4542-a6cd-31d353166cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:03:41.142282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:41.142282 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7460fa64-f0e9-4542-a6cd-31d353166cc1-maas-api-tls\") pod \"maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7\" (UID: \"7460fa64-f0e9-4542-a6cd-31d353166cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:03:41.142546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:41.142303 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7460fa64-f0e9-4542-a6cd-31d353166cc1-metrics-certs\") pod \"maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7\" (UID: \"7460fa64-f0e9-4542-a6cd-31d353166cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:03:41.142592 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:41.142535 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7460fa64-f0e9-4542-a6cd-31d353166cc1-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7\" (UID: \"7460fa64-f0e9-4542-a6cd-31d353166cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:03:41.243991 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:41.243894 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4z5h\" (UniqueName: \"kubernetes.io/projected/7460fa64-f0e9-4542-a6cd-31d353166cc1-kube-api-access-w4z5h\") pod \"maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7\" (UID: \"7460fa64-f0e9-4542-a6cd-31d353166cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:03:41.243991 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:41.243943 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7460fa64-f0e9-4542-a6cd-31d353166cc1-maas-api-tls\") pod \"maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7\" (UID: \"7460fa64-f0e9-4542-a6cd-31d353166cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:03:41.244240 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:41.244006 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7460fa64-f0e9-4542-a6cd-31d353166cc1-metrics-certs\") pod \"maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7\" (UID: \"7460fa64-f0e9-4542-a6cd-31d353166cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:03:41.244240 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:41.244091 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7460fa64-f0e9-4542-a6cd-31d353166cc1-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7\" (UID: \"7460fa64-f0e9-4542-a6cd-31d353166cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:03:41.244836 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:41.244806 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7460fa64-f0e9-4542-a6cd-31d353166cc1-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7\" (UID: \"7460fa64-f0e9-4542-a6cd-31d353166cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:03:41.246613 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:41.246590 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7460fa64-f0e9-4542-a6cd-31d353166cc1-maas-api-tls\") pod \"maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7\" (UID: \"7460fa64-f0e9-4542-a6cd-31d353166cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:03:41.246768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:41.246747 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7460fa64-f0e9-4542-a6cd-31d353166cc1-metrics-certs\") pod \"maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7\" (UID: \"7460fa64-f0e9-4542-a6cd-31d353166cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:03:41.267833 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:41.267797 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4z5h\" (UniqueName: \"kubernetes.io/projected/7460fa64-f0e9-4542-a6cd-31d353166cc1-kube-api-access-w4z5h\") pod \"maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7\" (UID: \"7460fa64-f0e9-4542-a6cd-31d353166cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:03:41.270646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:41.270617 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:03:41.577433 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:41.577344 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7"] Oct 30 13:03:41.580749 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:03:41.580721 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7460fa64_f0e9_4542_a6cd_31d353166cc1.slice/crio-205c89f8cfd6015c708b727c342a808434f3aa755e9f744d758698903899f4de WatchSource:0}: Error finding container 205c89f8cfd6015c708b727c342a808434f3aa755e9f744d758698903899f4de: Status 404 returned error can't find the container with id 205c89f8cfd6015c708b727c342a808434f3aa755e9f744d758698903899f4de Oct 30 13:03:42.364024 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:42.363947 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:03:42.382126 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:42.382090 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" event={"ID":"7460fa64-f0e9-4542-a6cd-31d353166cc1","Type":"ContainerStarted","Data":"d01290c689d595f75173c06b7d9f72862637a2b170739cb9336574b93807225b"} Oct 30 13:03:42.382272 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:42.382147 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" event={"ID":"7460fa64-f0e9-4542-a6cd-31d353166cc1","Type":"ContainerStarted","Data":"205c89f8cfd6015c708b727c342a808434f3aa755e9f744d758698903899f4de"} Oct 30 13:03:42.382272 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:42.382174 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:03:42.445477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:42.445424 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" podStartSLOduration=2.106841193 podStartE2EDuration="2.44540745s" totalImagesPullingTime="338.566257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:42.444563197 +0000 UTC m=+679.076750359" watchObservedRunningTime="2026-10-30 13:03:42.44540745 +0000 UTC m=+679.077594615" Oct 30 13:03:48.394591 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.394554 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:03:48.409152 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.409123 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq"] Oct 30 13:03:48.413302 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.413266 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.427838 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.427815 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-52a649\"/\"default-dockercfg-slxg9\"" Oct 30 13:03:48.431062 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.431043 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-52a649\"/\"openshift-service-ca.crt\"" Oct 30 13:03:48.431265 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.431246 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-52a649\"/\"kube-root-ca.crt\"" Oct 30 13:03:48.438943 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.438920 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-52a649\"/\"simulator-w1-52a649-kserve-self-signed-certs\"" Oct 30 13:03:48.440657 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.440633 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq"] Oct 30 13:03:48.501857 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.501826 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-dshm\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.502006 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.501884 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-model-cache\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.502006 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.501909 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-tls-certs\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.502108 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.502008 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-tmp-dir\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.502108 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.502029 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd75f\" (UniqueName: \"kubernetes.io/projected/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-kube-api-access-pd75f\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.502203 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.502164 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-home\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.603426 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.603381 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-model-cache\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.603426 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.603436 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-tls-certs\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.603718 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.603473 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-tmp-dir\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.603718 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.603632 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd75f\" (UniqueName: \"kubernetes.io/projected/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-kube-api-access-pd75f\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.603718 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.603698 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-home\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.603875 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.603808 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-dshm\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.603932 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.603885 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-model-cache\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.603988 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.603927 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-tmp-dir\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.604029 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.603995 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-home\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.605819 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.605791 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-dshm\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.605998 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.605982 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-tls-certs\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.641945 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.641907 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd75f\" (UniqueName: \"kubernetes.io/projected/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-kube-api-access-pd75f\") pod \"simulator-w1-52a649-kserve-5b9b5777-drfjq\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:48.727584 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:48.727473 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:49.041286 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:49.041172 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq"] Oct 30 13:03:49.045571 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:03:49.045538 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode56893d6_cf58_4ba8_b5b1_a5dc7780d7e9.slice/crio-44b18e5c4a0b39b917769c2bb1fc35f095ce963911a898802a795f0cea3f596a WatchSource:0}: Error finding container 44b18e5c4a0b39b917769c2bb1fc35f095ce963911a898802a795f0cea3f596a: Status 404 returned error can't find the container with id 44b18e5c4a0b39b917769c2bb1fc35f095ce963911a898802a795f0cea3f596a Oct 30 13:03:49.419783 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:49.419747 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" event={"ID":"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9","Type":"ContainerStarted","Data":"44b18e5c4a0b39b917769c2bb1fc35f095ce963911a898802a795f0cea3f596a"} Oct 30 13:03:50.426043 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:50.425960 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" event={"ID":"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9","Type":"ContainerStarted","Data":"6d8ebf7def231cd8c2263915af76780699089fb6ea2c591841c49117fe7afae3"} Oct 30 13:03:50.473628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:50.473569 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podStartSLOduration=1.37214733 podStartE2EDuration="2.473554669s" totalImagesPullingTime="1.101407339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:50.472417357 +0000 UTC m=+687.104604544" watchObservedRunningTime="2026-10-30 13:03:50.473554669 +0000 UTC m=+687.105741868" Oct 30 13:03:50.728149 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:50.728062 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:58.728421 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:58.728381 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:58.740939 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:58.740909 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:03:59.399725 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:03:59.399688 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:03:59.488081 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:03:59.488046 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:04:00.779976 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.779936 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt"] Oct 30 13:04:00.784246 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.784226 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.808867 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.808838 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ee7d65\"/\"openshift-service-ca.crt\"" Oct 30 13:04:00.808867 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.808857 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ee7d65\"/\"simulator-w6-ee7d65-kserve-self-signed-certs\"" Oct 30 13:04:00.809064 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.808836 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ee7d65\"/\"default-dockercfg-bn9k7\"" Oct 30 13:04:00.809064 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.808869 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ee7d65\"/\"kube-root-ca.crt\"" Oct 30 13:04:00.817384 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.812058 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt"] Oct 30 13:04:00.821632 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.821603 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7b1c88-2656-48b9-90ef-ec21b946341d-tls-certs\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.821740 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.821669 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq8nl\" (UniqueName: \"kubernetes.io/projected/ec7b1c88-2656-48b9-90ef-ec21b946341d-kube-api-access-fq8nl\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.821795 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.821762 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-home\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.821842 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.821793 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-model-cache\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.821956 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.821939 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-tmp-dir\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.822016 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.821978 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-dshm\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.922623 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.922580 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-tmp-dir\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.922623 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.922625 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-dshm\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.922889 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.922652 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7b1c88-2656-48b9-90ef-ec21b946341d-tls-certs\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.922889 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.922697 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq8nl\" (UniqueName: \"kubernetes.io/projected/ec7b1c88-2656-48b9-90ef-ec21b946341d-kube-api-access-fq8nl\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.922889 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.922725 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-home\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.922889 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.922750 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-model-cache\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.923193 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.923154 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-tmp-dir\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.923316 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.923207 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-home\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.923316 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.923236 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-model-cache\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.925098 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.925068 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-dshm\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.925411 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.925391 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7b1c88-2656-48b9-90ef-ec21b946341d-tls-certs\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:00.970008 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:00.969971 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq8nl\" (UniqueName: \"kubernetes.io/projected/ec7b1c88-2656-48b9-90ef-ec21b946341d-kube-api-access-fq8nl\") pod \"simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:01.096707 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:01.096615 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:01.399242 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:01.399207 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt"] Oct 30 13:04:01.404234 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:04:01.404205 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec7b1c88_2656_48b9_90ef_ec21b946341d.slice/crio-7bca4d32be237ddf03df8b2a999ba296e1957c7316425d2c9a2a699207d945c8 WatchSource:0}: Error finding container 7bca4d32be237ddf03df8b2a999ba296e1957c7316425d2c9a2a699207d945c8: Status 404 returned error can't find the container with id 7bca4d32be237ddf03df8b2a999ba296e1957c7316425d2c9a2a699207d945c8 Oct 30 13:04:01.488267 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:01.488226 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" event={"ID":"ec7b1c88-2656-48b9-90ef-ec21b946341d","Type":"ContainerStarted","Data":"aefc36da9e41ab8814e2e5c444902269bea81da46665b63c48f2bb714d975749"} Oct 30 13:04:01.488447 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:01.488279 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" event={"ID":"ec7b1c88-2656-48b9-90ef-ec21b946341d","Type":"ContainerStarted","Data":"7bca4d32be237ddf03df8b2a999ba296e1957c7316425d2c9a2a699207d945c8"} Oct 30 13:04:01.526682 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:01.526630 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podStartSLOduration=1.526614059 podStartE2EDuration="1.526614059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:01.525136375 +0000 UTC m=+698.157323538" watchObservedRunningTime="2026-10-30 13:04:01.526614059 +0000 UTC m=+698.158801222" Oct 30 13:04:02.096971 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:02.096935 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:11.097460 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:11.097419 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:11.110118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:11.110091 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:11.543928 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:11.543898 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:04:13.552501 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.552453 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq"] Oct 30 13:04:13.556909 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.556885 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.583618 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.583583 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d833bd\"/\"openshift-service-ca.crt\"" Oct 30 13:04:13.583618 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.583614 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d833bd\"/\"kube-root-ca.crt\"" Oct 30 13:04:13.583618 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.583627 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d833bd\"/\"default-dockercfg-xwlf5\"" Oct 30 13:04:13.583905 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.583678 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d833bd\"/\"simulator-w3-d833bd-kserve-self-signed-certs\"" Oct 30 13:04:13.585343 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.585317 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq"] Oct 30 13:04:13.651452 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.651403 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-dshm\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.651693 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.651464 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-model-cache\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.651693 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.651564 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-tmp-dir\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.651806 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.651709 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzwts\" (UniqueName: \"kubernetes.io/projected/b3640a4c-2e41-4186-ae55-d7a03c986793-kube-api-access-pzwts\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.651806 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.651739 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-home\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.651806 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.651759 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3640a4c-2e41-4186-ae55-d7a03c986793-tls-certs\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.753116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.753071 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzwts\" (UniqueName: \"kubernetes.io/projected/b3640a4c-2e41-4186-ae55-d7a03c986793-kube-api-access-pzwts\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.753341 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.753123 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-home\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.753341 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.753151 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3640a4c-2e41-4186-ae55-d7a03c986793-tls-certs\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.753341 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.753216 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-dshm\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.753341 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.753255 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-model-cache\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.753341 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.753293 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-tmp-dir\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.753721 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.753685 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-home\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.753789 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.753741 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-tmp-dir\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.753843 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.753796 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-model-cache\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.756285 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.756122 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-dshm\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.756540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.756513 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3640a4c-2e41-4186-ae55-d7a03c986793-tls-certs\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.778174 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.778142 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzwts\" (UniqueName: \"kubernetes.io/projected/b3640a4c-2e41-4186-ae55-d7a03c986793-kube-api-access-pzwts\") pod \"simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:13.870911 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:13.870885 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:14.180281 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:14.180150 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq"] Oct 30 13:04:14.185972 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:04:14.185936 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3640a4c_2e41_4186_ae55_d7a03c986793.slice/crio-5026ef693f123cf78a0c058e59db3a1ef05e2390f4260abc9dc99f043bbb57f5 WatchSource:0}: Error finding container 5026ef693f123cf78a0c058e59db3a1ef05e2390f4260abc9dc99f043bbb57f5: Status 404 returned error can't find the container with id 5026ef693f123cf78a0c058e59db3a1ef05e2390f4260abc9dc99f043bbb57f5 Oct 30 13:04:14.549619 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:14.549536 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" event={"ID":"b3640a4c-2e41-4186-ae55-d7a03c986793","Type":"ContainerStarted","Data":"8e6091a0468e8188798e496ee01e4568f3c4077453ff8f28603063277c5a711e"} Oct 30 13:04:14.549619 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:14.549587 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" event={"ID":"b3640a4c-2e41-4186-ae55-d7a03c986793","Type":"ContainerStarted","Data":"5026ef693f123cf78a0c058e59db3a1ef05e2390f4260abc9dc99f043bbb57f5"} Oct 30 13:04:14.638087 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:14.638031 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podStartSLOduration=1.638015535 podStartE2EDuration="1.638015535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:14.636758609 +0000 UTC m=+711.268945771" watchObservedRunningTime="2026-10-30 13:04:14.638015535 +0000 UTC m=+711.270202675" Oct 30 13:04:14.871684 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:14.871647 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:23.871313 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:23.871272 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:23.884216 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:23.884184 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:24.608377 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.608348 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:04:24.617744 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.617714 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2"] Oct 30 13:04:24.625313 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.625289 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.642356 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.642325 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2"] Oct 30 13:04:24.646142 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.646113 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d7e62b\"/\"openshift-service-ca.crt\"" Oct 30 13:04:24.646142 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.646114 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d7e62b\"/\"default-dockercfg-x76nv\"" Oct 30 13:04:24.646329 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.646173 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d7e62b\"/\"kube-root-ca.crt\"" Oct 30 13:04:24.652030 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.652012 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d7e62b\"/\"simulator-w4-d7e62b-kserve-self-signed-certs\"" Oct 30 13:04:24.767929 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.767890 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-dshm\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.768118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.767949 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-tmp-dir\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.768118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.768052 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3264707d-9edc-4a4e-976f-c4d3aed728b2-tls-certs\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.768118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.768089 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6m96\" (UniqueName: \"kubernetes.io/projected/3264707d-9edc-4a4e-976f-c4d3aed728b2-kube-api-access-l6m96\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.768282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.768179 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-home\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.768336 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.768290 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-model-cache\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.868979 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.868947 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-dshm\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.869165 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.869010 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-tmp-dir\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.869165 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.869054 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3264707d-9edc-4a4e-976f-c4d3aed728b2-tls-certs\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.869165 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.869081 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6m96\" (UniqueName: \"kubernetes.io/projected/3264707d-9edc-4a4e-976f-c4d3aed728b2-kube-api-access-l6m96\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.869165 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.869126 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-home\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.869398 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.869199 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-model-cache\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.869631 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.869605 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-home\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.869767 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.869661 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-model-cache\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.869835 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.869799 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-tmp-dir\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.872198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.872174 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3264707d-9edc-4a4e-976f-c4d3aed728b2-tls-certs\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.872617 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.872344 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-dshm\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.891814 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.891779 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6m96\" (UniqueName: \"kubernetes.io/projected/3264707d-9edc-4a4e-976f-c4d3aed728b2-kube-api-access-l6m96\") pod \"simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:24.938022 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:24.937979 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:25.250427 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:25.250333 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2"] Oct 30 13:04:25.255015 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:04:25.254986 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3264707d_9edc_4a4e_976f_c4d3aed728b2.slice/crio-d202ccb3beb01ecfb0163144ed7446157bd7efe89ae8cfc0084180b916e1c890 WatchSource:0}: Error finding container d202ccb3beb01ecfb0163144ed7446157bd7efe89ae8cfc0084180b916e1c890: Status 404 returned error can't find the container with id d202ccb3beb01ecfb0163144ed7446157bd7efe89ae8cfc0084180b916e1c890 Oct 30 13:04:25.603753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:25.603648 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" event={"ID":"3264707d-9edc-4a4e-976f-c4d3aed728b2","Type":"ContainerStarted","Data":"17dab731fefe765db2ad6b1ff9ec1ed23cf0aa488e5ce44c3af92548882e11d1"} Oct 30 13:04:25.603753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:25.603705 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" event={"ID":"3264707d-9edc-4a4e-976f-c4d3aed728b2","Type":"ContainerStarted","Data":"d202ccb3beb01ecfb0163144ed7446157bd7efe89ae8cfc0084180b916e1c890"} Oct 30 13:04:25.643974 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:25.643916 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podStartSLOduration=1.64390095 podStartE2EDuration="1.64390095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:25.643284566 +0000 UTC m=+722.275471758" watchObservedRunningTime="2026-10-30 13:04:25.64390095 +0000 UTC m=+722.276088117" Oct 30 13:04:25.939055 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:25.939016 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:29.405237 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:04:29.405194 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:04:33.911391 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:33.911356 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4"] Oct 30 13:04:33.915636 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:33.915609 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:33.931101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:33.931077 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e34fd5\"/\"simulator-w5-e34fd5-kserve-self-signed-certs\"" Oct 30 13:04:33.942242 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:33.942193 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e34fd5\"/\"kube-root-ca.crt\"" Oct 30 13:04:33.942542 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:33.942519 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e34fd5\"/\"openshift-service-ca.crt\"" Oct 30 13:04:33.944011 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:33.943985 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4"] Oct 30 13:04:33.953867 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:33.953839 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-tmp-dir\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:33.953965 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:33.953917 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-dshm\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:33.954009 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:33.953959 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-tls-certs\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:33.954076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:33.954054 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdlhs\" (UniqueName: \"kubernetes.io/projected/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-kube-api-access-pdlhs\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:33.954160 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:33.954149 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-model-cache\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:33.954195 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:33.954180 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-home\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:34.031500 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.031446 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e34fd5\"/\"default-dockercfg-vgqrm\"" Oct 30 13:04:34.055748 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.055651 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-dshm\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:34.055748 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.055696 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-tls-certs\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:34.055991 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.055750 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdlhs\" (UniqueName: \"kubernetes.io/projected/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-kube-api-access-pdlhs\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:34.055991 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.055823 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-model-cache\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:34.055991 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.055855 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-home\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:34.055991 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.055911 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-tmp-dir\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:34.056397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.056373 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-tmp-dir\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:34.056505 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.056395 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-home\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:34.056552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.056481 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-model-cache\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:34.058364 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.058339 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-tls-certs\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:34.058502 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.058472 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-dshm\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:34.088593 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.088550 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdlhs\" (UniqueName: \"kubernetes.io/projected/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-kube-api-access-pdlhs\") pod \"simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:34.229566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.229440 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:34.532322 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.532230 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4"] Oct 30 13:04:34.537341 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:04:34.537313 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8ceac9f_ae34_46f7_b04d_7d1484f880d7.slice/crio-ee816364acf60d9a31c52f51c5dc7f7a678acf50771f62089a30b0470dbefa84 WatchSource:0}: Error finding container ee816364acf60d9a31c52f51c5dc7f7a678acf50771f62089a30b0470dbefa84: Status 404 returned error can't find the container with id ee816364acf60d9a31c52f51c5dc7f7a678acf50771f62089a30b0470dbefa84 Oct 30 13:04:34.652100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.652060 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" event={"ID":"b8ceac9f-ae34-46f7-b04d-7d1484f880d7","Type":"ContainerStarted","Data":"9c3a7e26342b79b16ef4a54c56e15e5681d27a802447cffcaaa9867bc46408e7"} Oct 30 13:04:34.652274 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.652117 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" event={"ID":"b8ceac9f-ae34-46f7-b04d-7d1484f880d7","Type":"ContainerStarted","Data":"ee816364acf60d9a31c52f51c5dc7f7a678acf50771f62089a30b0470dbefa84"} Oct 30 13:04:34.693995 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.693862 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podStartSLOduration=1.693843244 podStartE2EDuration="1.693843244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:34.692001981 +0000 UTC m=+731.324189143" watchObservedRunningTime="2026-10-30 13:04:34.693843244 +0000 UTC m=+731.326030406" Oct 30 13:04:34.938263 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.938225 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:34.951231 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:34.951200 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:35.230363 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:35.230265 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:35.668600 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:35.668566 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:04:41.221971 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.221931 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp"] Oct 30 13:04:41.225650 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.225629 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.253695 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.253664 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ee7d65\"/\"premium-w6-ee7d65-kserve-self-signed-certs\"" Oct 30 13:04:41.256223 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.255981 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp"] Oct 30 13:04:41.330138 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.330105 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdgqk\" (UniqueName: \"kubernetes.io/projected/f1940153-057c-44d6-8117-c8746fa9d976-kube-api-access-kdgqk\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.330138 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.330140 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-dshm\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.330422 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.330288 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-home\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.330422 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.330371 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1940153-057c-44d6-8117-c8746fa9d976-tls-certs\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.330422 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.330403 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-model-cache\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.330620 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.330547 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-tmp-dir\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.432511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.432432 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-tmp-dir\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.432713 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.432537 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdgqk\" (UniqueName: \"kubernetes.io/projected/f1940153-057c-44d6-8117-c8746fa9d976-kube-api-access-kdgqk\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.432713 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.432575 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-dshm\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.432713 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.432628 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-home\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.432713 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.432693 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1940153-057c-44d6-8117-c8746fa9d976-tls-certs\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.432931 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.432738 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-model-cache\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.433200 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.433172 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-model-cache\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.434012 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.433983 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-tmp-dir\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.434286 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.434266 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-home\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.439656 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.437558 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1940153-057c-44d6-8117-c8746fa9d976-tls-certs\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.449603 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.449566 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-dshm\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.456314 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.456281 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdgqk\" (UniqueName: \"kubernetes.io/projected/f1940153-057c-44d6-8117-c8746fa9d976-kube-api-access-kdgqk\") pod \"premium-w6-ee7d65-kserve-97b8977fb-fr8lp\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.538925 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.538830 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:41.850795 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:41.850694 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp"] Oct 30 13:04:41.855181 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:04:41.855151 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1940153_057c_44d6_8117_c8746fa9d976.slice/crio-aabb19ad753912431b6b29a58abd17f7ff12294e590a8703f00046552779777a WatchSource:0}: Error finding container aabb19ad753912431b6b29a58abd17f7ff12294e590a8703f00046552779777a: Status 404 returned error can't find the container with id aabb19ad753912431b6b29a58abd17f7ff12294e590a8703f00046552779777a Oct 30 13:04:42.694396 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:42.694355 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" event={"ID":"f1940153-057c-44d6-8117-c8746fa9d976","Type":"ContainerStarted","Data":"ee67a97c7963452d1ad4014c4778f9c408b0e71fc4fac5c604a09e10563b0284"} Oct 30 13:04:42.694849 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:42.694410 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" event={"ID":"f1940153-057c-44d6-8117-c8746fa9d976","Type":"ContainerStarted","Data":"aabb19ad753912431b6b29a58abd17f7ff12294e590a8703f00046552779777a"} Oct 30 13:04:42.735880 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:42.735823 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podStartSLOduration=1.735803875 podStartE2EDuration="1.735803875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:42.733951791 +0000 UTC m=+739.366138953" watchObservedRunningTime="2026-10-30 13:04:42.735803875 +0000 UTC m=+739.367991037" Oct 30 13:04:43.538915 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:43.538880 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:44.229723 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:44.229688 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:44.242564 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:44.242533 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:44.715667 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:44.715637 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:04:51.539238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:51.539205 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:51.552255 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:51.552226 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:51.755853 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:04:51.755824 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:04:59.400691 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:04:59.400645 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:05:01.891217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:01.891168 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz"] Oct 30 13:05:01.895615 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:01.895593 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:01.912157 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:01.912128 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cb58b8\"/\"kube-root-ca.crt\"" Oct 30 13:05:01.925312 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:01.925268 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cb58b8\"/\"default-dockercfg-z4nxf\"" Oct 30 13:05:01.925312 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:01.925309 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cb58b8\"/\"premium-w2-cb58b8-kserve-self-signed-certs\"" Oct 30 13:05:01.926983 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:01.926940 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz"] Oct 30 13:05:02.011689 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.011656 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cb58b8\"/\"openshift-service-ca.crt\"" Oct 30 13:05:02.050084 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.050043 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f9a281c-7ca6-4157-9f9a-e86363233dcb-tls-certs\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.050293 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.050090 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-dshm\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.050293 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.050182 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-tmp-dir\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.050293 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.050238 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-home\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.050467 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.050354 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-model-cache\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.050467 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.050382 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-752v8\" (UniqueName: \"kubernetes.io/projected/7f9a281c-7ca6-4157-9f9a-e86363233dcb-kube-api-access-752v8\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.151462 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.151353 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-tmp-dir\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.151462 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.151417 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-home\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.151462 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.151459 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-model-cache\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.151783 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.151477 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-752v8\" (UniqueName: \"kubernetes.io/projected/7f9a281c-7ca6-4157-9f9a-e86363233dcb-kube-api-access-752v8\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.151783 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.151537 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f9a281c-7ca6-4157-9f9a-e86363233dcb-tls-certs\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.151783 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.151567 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-dshm\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.151934 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.151890 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-tmp-dir\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.151990 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.151960 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-home\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.152045 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.152001 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-model-cache\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.154267 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.154240 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-dshm\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.154465 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.154444 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f9a281c-7ca6-4157-9f9a-e86363233dcb-tls-certs\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.174565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.174531 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-752v8\" (UniqueName: \"kubernetes.io/projected/7f9a281c-7ca6-4157-9f9a-e86363233dcb-kube-api-access-752v8\") pod \"premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.210031 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.209988 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:02.520742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.520649 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz"] Oct 30 13:05:02.524508 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:05:02.524469 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f9a281c_7ca6_4157_9f9a_e86363233dcb.slice/crio-a33c93b8d9343a123eb853892d975de724558cad202fe90452ce0be633c491f2 WatchSource:0}: Error finding container a33c93b8d9343a123eb853892d975de724558cad202fe90452ce0be633c491f2: Status 404 returned error can't find the container with id a33c93b8d9343a123eb853892d975de724558cad202fe90452ce0be633c491f2 Oct 30 13:05:02.798897 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.798796 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" event={"ID":"7f9a281c-7ca6-4157-9f9a-e86363233dcb","Type":"ContainerStarted","Data":"153a75c9201cafe5c1f9951efd2043df419b0080154f7c062ed91e31f35a7dc7"} Oct 30 13:05:02.798897 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.798840 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" event={"ID":"7f9a281c-7ca6-4157-9f9a-e86363233dcb","Type":"ContainerStarted","Data":"a33c93b8d9343a123eb853892d975de724558cad202fe90452ce0be633c491f2"} Oct 30 13:05:02.839681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:02.839628 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podStartSLOduration=1.83961195 podStartE2EDuration="1.83961195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:02.838717824 +0000 UTC m=+759.470904989" watchObservedRunningTime="2026-10-30 13:05:02.83961195 +0000 UTC m=+759.471799111" Oct 30 13:05:03.210812 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:03.210777 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:12.210601 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:12.210557 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:12.223562 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:12.223535 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:12.857599 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:12.857560 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:05:14.526236 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.526192 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk"] Oct 30 13:05:14.530317 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.530293 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.555032 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.554995 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e34fd5\"/\"premium-w5-e34fd5-kserve-self-signed-certs\"" Oct 30 13:05:14.557244 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.557215 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk"] Oct 30 13:05:14.579004 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.578971 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb9ww\" (UniqueName: \"kubernetes.io/projected/7393071a-3009-43b4-afad-0ba041babef4-kube-api-access-jb9ww\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.579170 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.579015 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-dshm\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.579170 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.579040 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7393071a-3009-43b4-afad-0ba041babef4-tls-certs\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.579241 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.579155 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-tmp-dir\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.579241 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.579198 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-home\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.579303 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.579240 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-model-cache\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.680157 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.680117 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-dshm\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.680362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.680165 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7393071a-3009-43b4-afad-0ba041babef4-tls-certs\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.680362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.680190 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-tmp-dir\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.680362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.680218 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-home\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.680362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.680244 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-model-cache\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.681175 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.680413 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb9ww\" (UniqueName: \"kubernetes.io/projected/7393071a-3009-43b4-afad-0ba041babef4-kube-api-access-jb9ww\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.681175 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.681099 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-tmp-dir\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.681301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.681181 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-model-cache\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.681301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.681202 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-home\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.682738 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.682711 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-dshm\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.683026 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.683004 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7393071a-3009-43b4-afad-0ba041babef4-tls-certs\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.704274 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.704239 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb9ww\" (UniqueName: \"kubernetes.io/projected/7393071a-3009-43b4-afad-0ba041babef4-kube-api-access-jb9ww\") pod \"premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:14.845178 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:14.845083 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:15.154380 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:15.154349 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk"] Oct 30 13:05:15.158328 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:05:15.158290 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7393071a_3009_43b4_afad_0ba041babef4.slice/crio-2c21bb674bbf417906446e9c658c11d62ca67f304546d15e346131299aca1090 WatchSource:0}: Error finding container 2c21bb674bbf417906446e9c658c11d62ca67f304546d15e346131299aca1090: Status 404 returned error can't find the container with id 2c21bb674bbf417906446e9c658c11d62ca67f304546d15e346131299aca1090 Oct 30 13:05:15.867875 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:15.867830 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" event={"ID":"7393071a-3009-43b4-afad-0ba041babef4","Type":"ContainerStarted","Data":"723b5951e3d67519bbe9b7a3111ac0a12aa965282675e1d04a9e85dc7f00890f"} Oct 30 13:05:15.868289 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:15.867886 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" event={"ID":"7393071a-3009-43b4-afad-0ba041babef4","Type":"ContainerStarted","Data":"2c21bb674bbf417906446e9c658c11d62ca67f304546d15e346131299aca1090"} Oct 30 13:05:16.845864 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:16.845829 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:24.845317 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:24.845268 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:24.862016 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:24.861985 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:24.902961 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:24.901597 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podStartSLOduration=10.901582347 podStartE2EDuration="10.901582347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:15.915418942 +0000 UTC m=+772.547606104" watchObservedRunningTime="2026-10-30 13:05:24.901582347 +0000 UTC m=+781.533769510" Oct 30 13:05:24.925292 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:24.925256 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:05:29.401922 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:05:29.401873 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:05:52.433951 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.433917 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv"] Oct 30 13:05:52.439470 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.439435 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.464030 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.463995 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv"] Oct 30 13:05:52.467450 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.467425 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ee7d65-openshift-default-dockercfg-5cfmj\"" Oct 30 13:05:52.575857 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.575825 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.575857 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.575861 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.576057 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.575915 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.576097 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.576065 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.576132 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.576109 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.576262 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.576246 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.576349 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.576325 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.576503 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.576471 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.576641 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.576619 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4trs\" (UniqueName: \"kubernetes.io/projected/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-kube-api-access-q4trs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.677762 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.677726 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.677762 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.677767 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.678015 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.677807 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.678015 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.677832 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4trs\" (UniqueName: \"kubernetes.io/projected/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-kube-api-access-q4trs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.678015 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.677859 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.678015 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.677882 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.678015 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.677910 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.678015 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.677950 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.678015 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.677984 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.678378 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.678184 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.678378 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.678222 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.678526 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.678378 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.678526 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.678472 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.678731 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.678712 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.680201 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.680172 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.680380 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.680363 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.700016 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.699963 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4trs\" (UniqueName: \"kubernetes.io/projected/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-kube-api-access-q4trs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.700111 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.700095 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:52.754246 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:52.754219 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:53.075948 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:53.075864 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv"] Oct 30 13:05:53.079985 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:05:53.079953 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48852ae1_935d_4cfc_acd4_e2082a6ad5ab.slice/crio-8af16105b8d8e68bd47cd50ae060cabe2018eba306454330ac4ef8c611d3bfdd WatchSource:0}: Error finding container 8af16105b8d8e68bd47cd50ae060cabe2018eba306454330ac4ef8c611d3bfdd: Status 404 returned error can't find the container with id 8af16105b8d8e68bd47cd50ae060cabe2018eba306454330ac4ef8c611d3bfdd Oct 30 13:05:53.082303 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:53.082269 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:53.082382 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:53.082336 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:53.082382 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:53.082369 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:05:54.057992 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:54.057952 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" event={"ID":"48852ae1-935d-4cfc-acd4-e2082a6ad5ab","Type":"ContainerStarted","Data":"52bc9fda63c1c215fdd284723c6c152c056cd5ef9734269f4f3b9d17c3b49df5"} Oct 30 13:05:54.058391 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:54.058007 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" event={"ID":"48852ae1-935d-4cfc-acd4-e2082a6ad5ab","Type":"ContainerStarted","Data":"8af16105b8d8e68bd47cd50ae060cabe2018eba306454330ac4ef8c611d3bfdd"} Oct 30 13:05:54.112640 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:54.112574 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" podStartSLOduration=2.112555633 podStartE2EDuration="2.112555633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:54.111710791 +0000 UTC m=+810.743897973" watchObservedRunningTime="2026-10-30 13:05:54.112555633 +0000 UTC m=+810.744742795" Oct 30 13:05:54.755115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:54.755088 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:05:54.756299 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:54.756279 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 13:05:54.756386 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:54.756321 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" podUID="48852ae1-935d-4cfc-acd4-e2082a6ad5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 13:05:55.754506 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:55.754466 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 13:05:55.754903 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:55.754541 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" podUID="48852ae1-935d-4cfc-acd4-e2082a6ad5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 13:05:56.755200 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:56.755166 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 13:05:56.755614 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:56.755220 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" podUID="48852ae1-935d-4cfc-acd4-e2082a6ad5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 13:05:57.755475 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:57.755443 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 13:05:57.755889 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:57.755509 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" podUID="48852ae1-935d-4cfc-acd4-e2082a6ad5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 13:05:58.754890 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:58.754860 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 13:05:58.755059 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:58.754913 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" podUID="48852ae1-935d-4cfc-acd4-e2082a6ad5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 13:05:59.403311 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:05:59.403271 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:05:59.754848 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:59.754761 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 13:05:59.754848 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:05:59.754811 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" podUID="48852ae1-935d-4cfc-acd4-e2082a6ad5ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 13:06:00.759557 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:00.759529 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:06:00.759979 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:00.759792 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:06:00.760628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:00.760604 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:06:03.922203 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:03.922163 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct"] Oct 30 13:06:03.926512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:03.926474 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:03.953327 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:03.953250 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct"] Oct 30 13:06:04.020720 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.020695 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.020799 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.020741 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.020844 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.020826 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7824b283-1848-4c7c-a66c-a53c57354c0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.020890 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.020876 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.020992 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.020966 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.021097 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.021018 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.021097 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.021081 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnfkg\" (UniqueName: \"kubernetes.io/projected/7824b283-1848-4c7c-a66c-a53c57354c0d-kube-api-access-nnfkg\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.021171 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.021143 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.021171 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.021163 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.122711 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.122680 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.122884 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.122729 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.122884 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.122764 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.122884 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.122786 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnfkg\" (UniqueName: \"kubernetes.io/projected/7824b283-1848-4c7c-a66c-a53c57354c0d-kube-api-access-nnfkg\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.122884 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.122815 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.122884 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.122832 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.123180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.123006 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.123180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.123079 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.123180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.123125 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7824b283-1848-4c7c-a66c-a53c57354c0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.123342 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.123230 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.123417 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.123392 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.123529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.123464 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.123529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.123520 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.123918 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.123901 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7824b283-1848-4c7c-a66c-a53c57354c0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.125046 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.125027 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.125665 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.125634 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.146438 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.146403 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.146610 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.146580 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnfkg\" (UniqueName: \"kubernetes.io/projected/7824b283-1848-4c7c-a66c-a53c57354c0d-kube-api-access-nnfkg\") pod \"maas-default-gateway-openshift-default-64d4cc998c-vzfct\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.270257 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.270166 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:04.590281 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.590195 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct"] Oct 30 13:06:04.594178 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:06:04.594147 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7824b283_1848_4c7c_a66c_a53c57354c0d.slice/crio-6efca58ae19c07e0272ecab240d0c0677a3d50207a3da83142df6451fb200f48 WatchSource:0}: Error finding container 6efca58ae19c07e0272ecab240d0c0677a3d50207a3da83142df6451fb200f48: Status 404 returned error can't find the container with id 6efca58ae19c07e0272ecab240d0c0677a3d50207a3da83142df6451fb200f48 Oct 30 13:06:04.596498 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.596457 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:04.596584 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.596538 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:04.596584 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:04.596572 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:05.122550 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:05.122476 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" event={"ID":"7824b283-1848-4c7c-a66c-a53c57354c0d","Type":"ContainerStarted","Data":"a7f6ee046fe4bf16697f3ab832a9af3364035851899faac79628126c68598cbf"} Oct 30 13:06:05.122970 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:05.122575 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" event={"ID":"7824b283-1848-4c7c-a66c-a53c57354c0d","Type":"ContainerStarted","Data":"6efca58ae19c07e0272ecab240d0c0677a3d50207a3da83142df6451fb200f48"} Oct 30 13:06:05.166394 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:05.166343 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" podStartSLOduration=2.166329673 podStartE2EDuration="2.166329673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:05.16504304 +0000 UTC m=+821.797230215" watchObservedRunningTime="2026-10-30 13:06:05.166329673 +0000 UTC m=+821.798516834" Oct 30 13:06:05.270995 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:05.270957 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:05.272539 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:05.272513 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-vzfct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 13:06:05.272671 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:05.272575 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" podUID="7824b283-1848-4c7c-a66c-a53c57354c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 13:06:06.271452 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:06.271418 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-vzfct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 13:06:06.271848 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:06.271465 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" podUID="7824b283-1848-4c7c-a66c-a53c57354c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 13:06:07.271195 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.271162 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-vzfct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 13:06:07.271363 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.271225 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" podUID="7824b283-1848-4c7c-a66c-a53c57354c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 13:06:07.307510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.283872 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt"] Oct 30 13:06:07.312952 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.312925 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.313882 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.313392 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt"] Oct 30 13:06:07.327267 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.327241 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-52a649-openshift-default-dockercfg-frxr8\"" Oct 30 13:06:07.458401 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.458366 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbd94\" (UniqueName: \"kubernetes.io/projected/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-kube-api-access-pbd94\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.458683 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.458662 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.458791 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.458779 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.458937 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.458906 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.458999 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.458979 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.459051 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.459016 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.459106 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.459054 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.459159 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.459130 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.459215 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.459200 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.560763 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.560670 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.560763 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.560727 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.560984 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.560771 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.560984 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.560805 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.560984 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.560836 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.560984 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.560884 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.560984 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.560919 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.560984 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.560956 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.561273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.561027 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbd94\" (UniqueName: \"kubernetes.io/projected/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-kube-api-access-pbd94\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.561273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.561032 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.561273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.561057 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.561753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.561715 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.561886 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.561757 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.562030 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.562012 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.564747 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.564724 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.564865 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.564847 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.583167 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.583132 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbd94\" (UniqueName: \"kubernetes.io/projected/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-kube-api-access-pbd94\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.587519 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.583569 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:07.632217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:07.632188 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:08.014184 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.014146 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt"] Oct 30 13:06:08.016468 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:06:08.016441 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433f2adf_fc0f_4e5c_9bf3_056ab08e7891.slice/crio-9fe3ad92adc52e12b222bae8fa55c947a50bc0649e4ee208cf4543794cc959d5 WatchSource:0}: Error finding container 9fe3ad92adc52e12b222bae8fa55c947a50bc0649e4ee208cf4543794cc959d5: Status 404 returned error can't find the container with id 9fe3ad92adc52e12b222bae8fa55c947a50bc0649e4ee208cf4543794cc959d5 Oct 30 13:06:08.019258 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.019219 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:08.019328 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.019320 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:08.019369 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.019351 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:08.142728 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.142693 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" event={"ID":"433f2adf-fc0f-4e5c-9bf3-056ab08e7891","Type":"ContainerStarted","Data":"13a248e48ec3e6c460807a5bb88b48eade8c9ba0e00e238ffadbff5edc6ed63d"} Oct 30 13:06:08.142915 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.142749 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" event={"ID":"433f2adf-fc0f-4e5c-9bf3-056ab08e7891","Type":"ContainerStarted","Data":"9fe3ad92adc52e12b222bae8fa55c947a50bc0649e4ee208cf4543794cc959d5"} Oct 30 13:06:08.155598 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.155557 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p"] Oct 30 13:06:08.160097 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.160077 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.177430 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.177363 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f07dfe-openshift-default-dockercfg-d79w9\"" Oct 30 13:06:08.181532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.180182 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p"] Oct 30 13:06:08.252003 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.251924 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" podStartSLOduration=1.251903978 podStartE2EDuration="1.251903978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:08.207795336 +0000 UTC m=+824.839982500" watchObservedRunningTime="2026-10-30 13:06:08.251903978 +0000 UTC m=+824.884091141" Oct 30 13:06:08.269325 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.269227 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.269510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.269357 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9284f6a-9945-432a-be97-497fd1a027c5-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.269510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.269417 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9284f6a-9945-432a-be97-497fd1a027c5-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.269510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.269455 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr4l7\" (UniqueName: \"kubernetes.io/projected/e9284f6a-9945-432a-be97-497fd1a027c5-kube-api-access-pr4l7\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.269688 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.269481 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.269688 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.269609 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9284f6a-9945-432a-be97-497fd1a027c5-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.269688 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.269638 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.269848 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.269688 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.269848 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.269786 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.270483 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.270458 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-vzfct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 13:06:08.270612 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.270523 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" podUID="7824b283-1848-4c7c-a66c-a53c57354c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 13:06:08.371731 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.371693 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9284f6a-9945-432a-be97-497fd1a027c5-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.371731 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.371734 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.372278 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.371774 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.372278 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.371814 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.372278 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.371858 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.372278 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.371950 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9284f6a-9945-432a-be97-497fd1a027c5-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.372278 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.371988 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9284f6a-9945-432a-be97-497fd1a027c5-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.372278 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.372033 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr4l7\" (UniqueName: \"kubernetes.io/projected/e9284f6a-9945-432a-be97-497fd1a027c5-kube-api-access-pr4l7\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.372278 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.372060 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.372278 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.372191 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.372278 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.372230 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.372732 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.372431 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.372732 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.372712 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9284f6a-9945-432a-be97-497fd1a027c5-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.372940 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.372923 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.374775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.374754 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9284f6a-9945-432a-be97-497fd1a027c5-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.374870 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.374757 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.427085 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.427053 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr4l7\" (UniqueName: \"kubernetes.io/projected/e9284f6a-9945-432a-be97-497fd1a027c5-kube-api-access-pr4l7\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.443710 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.443682 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9284f6a-9945-432a-be97-497fd1a027c5-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.479529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.477466 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:08.632863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.632825 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:08.850607 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.850474 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p"] Oct 30 13:06:08.857870 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.857642 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:08.858695 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.858649 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:08.858806 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:08.858773 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:09.151695 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:09.151654 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" event={"ID":"e9284f6a-9945-432a-be97-497fd1a027c5","Type":"ContainerStarted","Data":"1672ae1231da0275d877b996ee7e2dc86d4704b3d93ea14ce79b651daca7b3de"} Oct 30 13:06:09.151906 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:09.151735 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" event={"ID":"e9284f6a-9945-432a-be97-497fd1a027c5","Type":"ContainerStarted","Data":"cdf1de633425a1781d468ac6e2692272488ce00bb573c50f83afe3a4918cac93"} Oct 30 13:06:09.270649 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:09.270609 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-vzfct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 13:06:09.270849 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:09.270667 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" podUID="7824b283-1848-4c7c-a66c-a53c57354c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 13:06:09.480326 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:09.478865 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:09.480712 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:09.480325 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:06:09.480712 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:09.480366 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" podUID="e9284f6a-9945-432a-be97-497fd1a027c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 13:06:09.642568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:09.638964 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:06:09.642568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:09.639023 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" podUID="433f2adf-fc0f-4e5c-9bf3-056ab08e7891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 13:06:10.271163 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:10.271125 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-vzfct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 13:06:10.271344 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:10.271176 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" podUID="7824b283-1848-4c7c-a66c-a53c57354c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 13:06:10.478975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:10.478923 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:06:10.479176 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:10.478991 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" podUID="e9284f6a-9945-432a-be97-497fd1a027c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 13:06:10.641513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:10.640905 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:06:10.641513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:10.640965 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" podUID="433f2adf-fc0f-4e5c-9bf3-056ab08e7891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 13:06:11.271372 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:11.271227 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-vzfct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 13:06:11.271372 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:11.271290 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" podUID="7824b283-1848-4c7c-a66c-a53c57354c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 13:06:11.480405 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:11.478994 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:06:11.480405 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:11.479041 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" podUID="e9284f6a-9945-432a-be97-497fd1a027c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 13:06:11.633515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:11.632675 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:06:11.633515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:11.632740 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" podUID="433f2adf-fc0f-4e5c-9bf3-056ab08e7891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 13:06:12.271056 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:12.271013 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-vzfct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 13:06:12.271445 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:12.271070 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" podUID="7824b283-1848-4c7c-a66c-a53c57354c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 13:06:12.478530 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:12.478500 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:06:12.478758 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:12.478739 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" podUID="e9284f6a-9945-432a-be97-497fd1a027c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 13:06:12.632947 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:12.632908 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:06:12.633238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:12.633206 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" podUID="433f2adf-fc0f-4e5c-9bf3-056ab08e7891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 13:06:13.478689 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:13.478659 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:06:13.479129 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:13.479106 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" podUID="e9284f6a-9945-432a-be97-497fd1a027c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 13:06:13.632971 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:13.632924 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:06:13.633245 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:13.633212 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" podUID="433f2adf-fc0f-4e5c-9bf3-056ab08e7891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 13:06:14.253804 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:14.253769 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:14.254685 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:14.254664 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:14.255826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:14.255809 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:06:14.342997 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:14.342936 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" podStartSLOduration=6.342916852 podStartE2EDuration="6.342916852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:09.205109003 +0000 UTC m=+825.837296166" watchObservedRunningTime="2026-10-30 13:06:14.342916852 +0000 UTC m=+830.975104016" Oct 30 13:06:14.478594 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:14.478557 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:06:14.478766 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:14.478627 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" podUID="e9284f6a-9945-432a-be97-497fd1a027c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 13:06:14.632999 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:14.632966 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 13:06:14.633175 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:14.633028 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" podUID="433f2adf-fc0f-4e5c-9bf3-056ab08e7891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 13:06:15.478768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:15.478726 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 13:06:15.479239 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:15.478806 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" podUID="e9284f6a-9945-432a-be97-497fd1a027c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 13:06:15.636577 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:15.636542 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:15.636944 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:15.636916 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:15.637727 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:15.637709 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:06:16.511707 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:16.511676 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:16.512580 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:16.512544 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:16.513438 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:16.513415 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:06:18.955988 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:18.955952 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2"] Oct 30 13:06:18.960231 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:18.960209 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:18.989604 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:18.989571 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2"] Oct 30 13:06:19.081782 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.081739 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd79a34d-b24a-41f2-b01a-55b70b9972f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.081782 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.081782 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h2r5\" (UniqueName: \"kubernetes.io/projected/fd79a34d-b24a-41f2-b01a-55b70b9972f2-kube-api-access-4h2r5\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.082038 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.081890 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd79a34d-b24a-41f2-b01a-55b70b9972f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.082038 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.081941 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd79a34d-b24a-41f2-b01a-55b70b9972f2-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.082038 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.082005 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd79a34d-b24a-41f2-b01a-55b70b9972f2-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.082156 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.082039 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd79a34d-b24a-41f2-b01a-55b70b9972f2-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.082156 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.082072 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd79a34d-b24a-41f2-b01a-55b70b9972f2-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.082220 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.082154 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd79a34d-b24a-41f2-b01a-55b70b9972f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.082263 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.082217 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd79a34d-b24a-41f2-b01a-55b70b9972f2-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.182828 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.182790 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h2r5\" (UniqueName: \"kubernetes.io/projected/fd79a34d-b24a-41f2-b01a-55b70b9972f2-kube-api-access-4h2r5\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.183008 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.182844 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd79a34d-b24a-41f2-b01a-55b70b9972f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.183008 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.182873 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd79a34d-b24a-41f2-b01a-55b70b9972f2-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.183086 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.183002 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd79a34d-b24a-41f2-b01a-55b70b9972f2-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.183086 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.183049 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd79a34d-b24a-41f2-b01a-55b70b9972f2-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.183171 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.183104 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd79a34d-b24a-41f2-b01a-55b70b9972f2-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.183171 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.183159 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd79a34d-b24a-41f2-b01a-55b70b9972f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.183276 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.183236 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd79a34d-b24a-41f2-b01a-55b70b9972f2-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.183335 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.183282 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd79a34d-b24a-41f2-b01a-55b70b9972f2-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.183335 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.183310 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd79a34d-b24a-41f2-b01a-55b70b9972f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.183482 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.183461 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd79a34d-b24a-41f2-b01a-55b70b9972f2-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.183482 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.183475 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd79a34d-b24a-41f2-b01a-55b70b9972f2-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.183617 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.183576 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd79a34d-b24a-41f2-b01a-55b70b9972f2-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.183908 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.183891 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd79a34d-b24a-41f2-b01a-55b70b9972f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.185316 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.185300 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd79a34d-b24a-41f2-b01a-55b70b9972f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.185711 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.185692 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd79a34d-b24a-41f2-b01a-55b70b9972f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.206352 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.206269 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd79a34d-b24a-41f2-b01a-55b70b9972f2-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.206477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.206362 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h2r5\" (UniqueName: \"kubernetes.io/projected/fd79a34d-b24a-41f2-b01a-55b70b9972f2-kube-api-access-4h2r5\") pod \"maas-default-gateway-openshift-default-64d4cc998c-k68w2\" (UID: \"fd79a34d-b24a-41f2-b01a-55b70b9972f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.274979 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.274936 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:19.609743 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.609654 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2"] Oct 30 13:06:19.613972 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:06:19.613931 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd79a34d_b24a_41f2_b01a_55b70b9972f2.slice/crio-055ba29c06896776120eb9d8e89f8c629886dbe6c12011556402760ce61ae8f8 WatchSource:0}: Error finding container 055ba29c06896776120eb9d8e89f8c629886dbe6c12011556402760ce61ae8f8: Status 404 returned error can't find the container with id 055ba29c06896776120eb9d8e89f8c629886dbe6c12011556402760ce61ae8f8 Oct 30 13:06:19.616193 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.616163 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:19.616279 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.616226 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:19.616279 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:19.616254 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:20.253367 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:20.253325 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" event={"ID":"fd79a34d-b24a-41f2-b01a-55b70b9972f2","Type":"ContainerStarted","Data":"10e46ef0d2215facf26d2f1fe5b8d84c192ed46033a3d4cce46b22e96326a07f"} Oct 30 13:06:20.253798 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:20.253388 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" event={"ID":"fd79a34d-b24a-41f2-b01a-55b70b9972f2","Type":"ContainerStarted","Data":"055ba29c06896776120eb9d8e89f8c629886dbe6c12011556402760ce61ae8f8"} Oct 30 13:06:20.275934 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:20.275900 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:20.277403 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:20.277377 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-k68w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:06:20.277561 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:20.277430 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" podUID="fd79a34d-b24a-41f2-b01a-55b70b9972f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 13:06:21.275852 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:21.275820 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-k68w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:06:21.276344 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:21.275871 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" podUID="fd79a34d-b24a-41f2-b01a-55b70b9972f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 13:06:22.276157 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:22.276125 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-k68w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:06:22.276528 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:22.276173 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" podUID="fd79a34d-b24a-41f2-b01a-55b70b9972f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 13:06:23.276047 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.276012 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-k68w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:06:23.276251 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.276067 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" podUID="fd79a34d-b24a-41f2-b01a-55b70b9972f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 13:06:23.528213 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.528050 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" podStartSLOduration=5.528037169 podStartE2EDuration="5.528037169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:20.318016936 +0000 UTC m=+836.950204102" watchObservedRunningTime="2026-10-30 13:06:23.528037169 +0000 UTC m=+840.160224331" Oct 30 13:06:23.550483 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.528237 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk"] Oct 30 13:06:23.550483 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.532679 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.553829 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.553805 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d833bd-openshift-default-dockercfg-dpcql\"" Oct 30 13:06:23.564274 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.564247 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk"] Oct 30 13:06:23.648047 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.648009 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xl94\" (UniqueName: \"kubernetes.io/projected/27c0ae74-f815-4746-94d7-959ad7a2733e-kube-api-access-5xl94\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.648285 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.648264 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c0ae74-f815-4746-94d7-959ad7a2733e-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.648406 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.648390 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.648447 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.648430 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.648601 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.648578 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.648686 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.648634 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.648784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.648762 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.648869 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.648856 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.648945 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.648933 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.750261 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.750226 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c0ae74-f815-4746-94d7-959ad7a2733e-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.750436 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.750295 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.750436 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.750319 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.750436 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.750362 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.750436 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.750400 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.750436 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.750426 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.750645 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.750452 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.750645 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.750534 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.750645 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.750579 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xl94\" (UniqueName: \"kubernetes.io/projected/27c0ae74-f815-4746-94d7-959ad7a2733e-kube-api-access-5xl94\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.750799 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.750770 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.750985 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.750955 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c0ae74-f815-4746-94d7-959ad7a2733e-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.751121 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.751098 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.751121 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.751054 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.751233 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.750971 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.752697 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.752678 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.752948 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.752930 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.772599 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.772575 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.773067 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.773045 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xl94\" (UniqueName: \"kubernetes.io/projected/27c0ae74-f815-4746-94d7-959ad7a2733e-kube-api-access-5xl94\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:23.848369 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:23.848286 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:24.213755 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:24.213718 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk"] Oct 30 13:06:24.214592 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:06:24.214566 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c0ae74_f815_4746_94d7_959ad7a2733e.slice/crio-a344057e0069e18ca4fb722c47f3c61d7029414417653a459925bef14c488586 WatchSource:0}: Error finding container a344057e0069e18ca4fb722c47f3c61d7029414417653a459925bef14c488586: Status 404 returned error can't find the container with id a344057e0069e18ca4fb722c47f3c61d7029414417653a459925bef14c488586 Oct 30 13:06:24.217280 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:24.217247 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:24.217393 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:24.217322 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:24.217393 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:24.217373 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:24.275802 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:24.275775 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-k68w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:06:24.275922 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:24.275831 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" podUID="fd79a34d-b24a-41f2-b01a-55b70b9972f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 13:06:24.280891 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:24.280858 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" event={"ID":"27c0ae74-f815-4746-94d7-959ad7a2733e","Type":"ContainerStarted","Data":"a344057e0069e18ca4fb722c47f3c61d7029414417653a459925bef14c488586"} Oct 30 13:06:25.276805 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:25.275971 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-k68w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:06:25.276805 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:25.276021 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" podUID="fd79a34d-b24a-41f2-b01a-55b70b9972f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 13:06:25.301508 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:25.297382 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" event={"ID":"27c0ae74-f815-4746-94d7-959ad7a2733e","Type":"ContainerStarted","Data":"ea57c142625133eea709eba14b2cf672de303eb82f3f4f2506ebb5644e16a2b8"} Oct 30 13:06:25.358342 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:25.358001 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" podStartSLOduration=2.357982333 podStartE2EDuration="2.357982333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:25.356835123 +0000 UTC m=+841.989022283" watchObservedRunningTime="2026-10-30 13:06:25.357982333 +0000 UTC m=+841.990169510" Oct 30 13:06:25.849178 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:25.849145 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:25.850327 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:25.850290 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:06:25.850461 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:25.850357 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" podUID="27c0ae74-f815-4746-94d7-959ad7a2733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 13:06:26.275631 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:26.275586 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-k68w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:06:26.275812 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:26.275656 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" podUID="fd79a34d-b24a-41f2-b01a-55b70b9972f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 13:06:26.849252 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:26.849215 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:06:26.849771 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:26.849275 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" podUID="27c0ae74-f815-4746-94d7-959ad7a2733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 13:06:27.276203 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:27.276164 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-k68w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 13:06:27.276375 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:27.276228 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" podUID="fd79a34d-b24a-41f2-b01a-55b70b9972f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 13:06:27.849608 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:27.849574 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:06:27.850035 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:27.849632 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" podUID="27c0ae74-f815-4746-94d7-959ad7a2733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 13:06:28.848979 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:28.848943 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:06:28.849192 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:28.849001 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" podUID="27c0ae74-f815-4746-94d7-959ad7a2733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 13:06:28.915663 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:28.915631 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:28.916484 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:28.916459 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:28.917298 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:28.917274 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" Oct 30 13:06:29.420556 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:06:29.420516 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:06:29.848713 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:29.848628 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:06:29.848713 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:29.848693 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" podUID="27c0ae74-f815-4746-94d7-959ad7a2733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 13:06:30.849535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:30.849497 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 13:06:30.850054 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:30.849558 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" podUID="27c0ae74-f815-4746-94d7-959ad7a2733e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 13:06:31.853217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:31.853187 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:31.853652 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:31.853479 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:31.854270 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:31.854249 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:06:53.373078 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.373041 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw"] Oct 30 13:06:53.378441 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.378418 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.396501 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.396464 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-e34fd5-openshift-default-dockercfg-mz9s9\"" Oct 30 13:06:53.415206 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.411526 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw"] Oct 30 13:06:53.510344 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.510300 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.510583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.510362 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.510583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.510469 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.510583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.510548 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n95pl\" (UniqueName: \"kubernetes.io/projected/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-kube-api-access-n95pl\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.510771 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.510628 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.510771 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.510672 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.510771 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.510748 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.510927 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.510771 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.510927 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.510800 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.612050 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.612016 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.612242 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.612065 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.612242 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.612091 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.612242 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.612131 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.612242 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.612171 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.612242 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.612209 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.612573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.612270 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.612573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.612351 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n95pl\" (UniqueName: \"kubernetes.io/projected/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-kube-api-access-n95pl\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.612573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.612404 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.612573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.612411 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.612751 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.612593 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.612751 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.612639 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.612751 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.612663 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.613273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.613245 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.614533 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.614514 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.614969 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.614947 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.637902 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.637835 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.637902 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.637874 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n95pl\" (UniqueName: \"kubernetes.io/projected/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-kube-api-access-n95pl\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:53.695696 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:53.695664 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:54.026098 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:54.026069 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw"] Oct 30 13:06:54.028696 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:06:54.028668 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b36af0_e09d_4810_b85c_e5ed5e5e2ee6.slice/crio-76cfaf9f99f3c1cc25b329782881c6a07b3ab4008ac536b87c1969c6650b396d WatchSource:0}: Error finding container 76cfaf9f99f3c1cc25b329782881c6a07b3ab4008ac536b87c1969c6650b396d: Status 404 returned error can't find the container with id 76cfaf9f99f3c1cc25b329782881c6a07b3ab4008ac536b87c1969c6650b396d Oct 30 13:06:54.031198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:54.031168 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:54.031295 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:54.031230 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:54.031295 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:54.031262 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:06:54.479645 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:54.479603 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" event={"ID":"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6","Type":"ContainerStarted","Data":"5f1f0d55567654d8d74c71e7481622d713fdf03663d29f8d28f21c2bb1a47b22"} Oct 30 13:06:54.480037 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:54.479669 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" event={"ID":"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6","Type":"ContainerStarted","Data":"76cfaf9f99f3c1cc25b329782881c6a07b3ab4008ac536b87c1969c6650b396d"} Oct 30 13:06:54.565132 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:54.565070 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" podStartSLOduration=1.5650557219999999 podStartE2EDuration="1.565055722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:54.55075724 +0000 UTC m=+871.182944402" watchObservedRunningTime="2026-10-30 13:06:54.565055722 +0000 UTC m=+871.197242884" Oct 30 13:06:54.704628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:54.704584 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:06:55.704514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:55.703461 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 13:06:55.704514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:55.703558 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" podUID="84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 13:06:56.703106 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:56.702624 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 13:06:56.703106 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:56.702703 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" podUID="84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 13:06:57.700855 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:57.700820 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 13:06:57.701442 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:57.701411 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" podUID="84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 13:06:58.701911 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:58.701872 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 13:06:58.702690 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:58.702655 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" podUID="84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 13:06:59.421137 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:06:59.421094 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:06:59.702961 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:59.702870 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 13:06:59.703550 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:06:59.703518 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" podUID="84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 13:07:01.129036 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:01.129005 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:07:01.130454 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:01.130430 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:07:01.132288 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:01.132262 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:07:07.825688 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:07.825651 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk"] Oct 30 13:07:07.830777 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:07.830751 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:07.847114 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:07.847088 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d7e62b-openshift-default-dockercfg-29w5v\"" Oct 30 13:07:07.863128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:07.863098 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk"] Oct 30 13:07:07.903605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:07.903570 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:07.903757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:07.903618 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:07.903831 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:07.903752 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sq7m\" (UniqueName: \"kubernetes.io/projected/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-kube-api-access-6sq7m\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:07.903831 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:07.903796 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:07.903939 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:07.903886 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:07.904021 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:07.904003 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:07.904082 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:07.904061 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:07.904126 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:07.904092 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:07.904168 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:07.904130 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.004775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.004736 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sq7m\" (UniqueName: \"kubernetes.io/projected/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-kube-api-access-6sq7m\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.004975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.004795 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.004975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.004848 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.004975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.004922 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.004975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.004965 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.005249 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.004992 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.005249 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.005047 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.005249 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.005103 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.005249 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.005129 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.005553 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.005522 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.005670 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.005645 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.005770 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.005750 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.005824 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.005752 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.005876 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.005856 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.007616 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.007592 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.007938 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.007914 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.029025 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.028991 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.029146 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.029068 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sq7m\" (UniqueName: \"kubernetes.io/projected/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-kube-api-access-6sq7m\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.145634 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.145608 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:08.506869 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.506753 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk"] Oct 30 13:07:08.509307 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:07:08.509279 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2adf6e82_f5f4_48cc_aca3_85a34f5ec552.slice/crio-d6c106f0870a1e1e6dfb0d952fffc68e71a901fa9ca3998de966e11f542d9c81 WatchSource:0}: Error finding container d6c106f0870a1e1e6dfb0d952fffc68e71a901fa9ca3998de966e11f542d9c81: Status 404 returned error can't find the container with id d6c106f0870a1e1e6dfb0d952fffc68e71a901fa9ca3998de966e11f542d9c81 Oct 30 13:07:08.511448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.511416 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:08.511573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.511478 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:08.511573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.511540 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:08.579663 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.579631 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" event={"ID":"2adf6e82-f5f4-48cc-aca3-85a34f5ec552","Type":"ContainerStarted","Data":"6ebc71d3ea9ad9fbde49c8c7c2261aa68cb9ba8948f255306901bbcc1283b863"} Oct 30 13:07:08.579771 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.579694 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" event={"ID":"2adf6e82-f5f4-48cc-aca3-85a34f5ec552","Type":"ContainerStarted","Data":"d6c106f0870a1e1e6dfb0d952fffc68e71a901fa9ca3998de966e11f542d9c81"} Oct 30 13:07:08.642886 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:08.642819 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" podStartSLOduration=1.6427992310000001 podStartE2EDuration="1.642799231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:08.639311447 +0000 UTC m=+885.271498608" watchObservedRunningTime="2026-10-30 13:07:08.642799231 +0000 UTC m=+885.274986394" Oct 30 13:07:09.146281 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:09.146250 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:09.148048 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:09.148021 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 13:07:09.148171 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:09.148070 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" podUID="2adf6e82-f5f4-48cc-aca3-85a34f5ec552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 13:07:10.146104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:10.146071 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 13:07:10.146287 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:10.146137 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" podUID="2adf6e82-f5f4-48cc-aca3-85a34f5ec552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 13:07:11.146264 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:11.146232 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 13:07:11.146697 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:11.146288 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" podUID="2adf6e82-f5f4-48cc-aca3-85a34f5ec552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 13:07:12.146285 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:12.146242 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 13:07:12.146696 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:12.146310 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" podUID="2adf6e82-f5f4-48cc-aca3-85a34f5ec552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 13:07:13.146267 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:13.146232 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 13:07:13.146474 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:13.146311 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" podUID="2adf6e82-f5f4-48cc-aca3-85a34f5ec552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 13:07:14.146600 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:14.146567 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 13:07:14.146995 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:14.146620 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" podUID="2adf6e82-f5f4-48cc-aca3-85a34f5ec552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 13:07:15.146015 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:15.145969 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 13:07:15.146228 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:15.146048 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" podUID="2adf6e82-f5f4-48cc-aca3-85a34f5ec552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 13:07:16.912756 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:16.912722 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:16.913190 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:16.913109 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:16.914627 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:16.914603 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:07:23.589673 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:23.589638 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:07:23.593543 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:23.593516 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:07:29.401079 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:07:29.401016 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:07:53.587237 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.587161 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6"] Oct 30 13:07:53.591018 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.590998 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.616707 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.616673 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6"] Oct 30 13:07:53.659010 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.658981 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj"] Oct 30 13:07:53.662700 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.662682 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.700561 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.700529 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj"] Oct 30 13:07:53.783781 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.783743 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.783973 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.783805 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.783973 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.783884 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f58b078-73f0-4d86-a905-2a914d54687c-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.783973 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.783945 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.784120 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.784026 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.784120 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.784055 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.784120 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.784073 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.784120 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.784088 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f58b078-73f0-4d86-a905-2a914d54687c-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.784332 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.784201 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.784332 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.784254 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.784443 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.784333 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7fpk\" (UniqueName: \"kubernetes.io/projected/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-kube-api-access-h7fpk\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.784443 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.784395 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.784576 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.784446 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hbwt\" (UniqueName: \"kubernetes.io/projected/2f58b078-73f0-4d86-a905-2a914d54687c-kube-api-access-5hbwt\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.784576 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.784527 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.784576 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.784565 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.784718 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.784601 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.784718 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.784656 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.784718 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.784702 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f58b078-73f0-4d86-a905-2a914d54687c-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.885384 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.885353 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.885605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.885399 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f58b078-73f0-4d86-a905-2a914d54687c-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.885605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.885467 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.885605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.885542 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.885605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.885574 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.885605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.885599 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.885946 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.885623 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f58b078-73f0-4d86-a905-2a914d54687c-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.885946 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.885656 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.885946 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.885703 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.885946 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.885751 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7fpk\" (UniqueName: \"kubernetes.io/projected/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-kube-api-access-h7fpk\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.885946 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.885795 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.885946 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.885847 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hbwt\" (UniqueName: \"kubernetes.io/projected/2f58b078-73f0-4d86-a905-2a914d54687c-kube-api-access-5hbwt\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.885946 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.885897 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.885946 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.885936 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.886302 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.885949 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.886302 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.885961 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.886302 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.885997 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.886302 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.886001 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.886302 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.886066 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f58b078-73f0-4d86-a905-2a914d54687c-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.886302 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.886107 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.886302 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.886292 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.886620 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.886461 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.886972 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.886947 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.887073 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.886987 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f58b078-73f0-4d86-a905-2a914d54687c-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.887632 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.887584 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.887632 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.887607 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.888325 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.887861 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.888433 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.888409 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.888778 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.888759 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.888876 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.888801 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.889161 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.889137 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f58b078-73f0-4d86-a905-2a914d54687c-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.890851 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.890828 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.911024 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.910985 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f58b078-73f0-4d86-a905-2a914d54687c-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.911736 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.911714 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.912726 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.912706 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hbwt\" (UniqueName: \"kubernetes.io/projected/2f58b078-73f0-4d86-a905-2a914d54687c-kube-api-access-5hbwt\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:53.912982 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.912963 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7fpk\" (UniqueName: \"kubernetes.io/projected/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-kube-api-access-h7fpk\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:53.976345 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:53.976309 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:54.207129 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:54.207041 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:54.302520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:54.302476 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj"] Oct 30 13:07:54.309027 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:07:54.308999 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f58b078_73f0_4d86_a905_2a914d54687c.slice/crio-4fb8c0e09fa544fa8a1ebe96a867bf5c421bef326565c04619acecc47d79f6b9 WatchSource:0}: Error finding container 4fb8c0e09fa544fa8a1ebe96a867bf5c421bef326565c04619acecc47d79f6b9: Status 404 returned error can't find the container with id 4fb8c0e09fa544fa8a1ebe96a867bf5c421bef326565c04619acecc47d79f6b9 Oct 30 13:07:54.311946 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:54.311904 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:54.312049 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:54.311989 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:54.312049 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:54.312035 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:54.516137 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:54.516110 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6"] Oct 30 13:07:54.518698 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:07:54.518671 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7a5fa5_c1d8_4556_99f1_a0481dafc16d.slice/crio-d0bdb046011632c4237e406ad9dc0f430787ff122d2938e3132ef4ccd26da9a8 WatchSource:0}: Error finding container d0bdb046011632c4237e406ad9dc0f430787ff122d2938e3132ef4ccd26da9a8: Status 404 returned error can't find the container with id d0bdb046011632c4237e406ad9dc0f430787ff122d2938e3132ef4ccd26da9a8 Oct 30 13:07:54.521220 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:54.521190 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:54.521308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:54.521252 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:54.521308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:54.521279 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:07:54.875791 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:54.875752 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" event={"ID":"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d","Type":"ContainerStarted","Data":"9e3ee119c33c324fa4a1a97fd09e6ed29cedaa2c02f76649e64162494bf2498c"} Oct 30 13:07:54.876531 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:54.875823 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" event={"ID":"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d","Type":"ContainerStarted","Data":"d0bdb046011632c4237e406ad9dc0f430787ff122d2938e3132ef4ccd26da9a8"} Oct 30 13:07:54.877753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:54.877723 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" event={"ID":"2f58b078-73f0-4d86-a905-2a914d54687c","Type":"ContainerStarted","Data":"2044964a3702712a3ba59017befb51dd5b335c6c0f6ad6fd142c27a6884744ef"} Oct 30 13:07:54.877856 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:54.877777 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" event={"ID":"2f58b078-73f0-4d86-a905-2a914d54687c","Type":"ContainerStarted","Data":"4fb8c0e09fa544fa8a1ebe96a867bf5c421bef326565c04619acecc47d79f6b9"} Oct 30 13:07:54.930703 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:54.930651 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" podStartSLOduration=1.930634939 podStartE2EDuration="1.930634939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:54.927900257 +0000 UTC m=+931.560087419" watchObservedRunningTime="2026-10-30 13:07:54.930634939 +0000 UTC m=+931.562822101" Oct 30 13:07:54.977210 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:54.977175 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:07:54.982942 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:54.982895 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" podStartSLOduration=1.982880924 podStartE2EDuration="1.982880924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:54.981434842 +0000 UTC m=+931.613622003" watchObservedRunningTime="2026-10-30 13:07:54.982880924 +0000 UTC m=+931.615068085" Oct 30 13:07:55.208728 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:55.208582 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:07:55.209399 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:55.209368 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 13:07:55.209664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:55.209638 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" podUID="ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 13:07:55.977935 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:55.977892 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 13:07:55.978381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:55.978349 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" podUID="2f58b078-73f0-4d86-a905-2a914d54687c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 13:07:56.209517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:56.209082 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 13:07:56.209517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:56.209200 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" podUID="ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 13:07:56.977340 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:56.977306 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 13:07:56.977528 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:56.977366 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" podUID="2f58b078-73f0-4d86-a905-2a914d54687c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 13:07:57.234265 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:57.233716 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 13:07:57.234265 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:57.233776 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" podUID="ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 13:07:57.977204 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:57.977170 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 13:07:57.977353 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:57.977228 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" podUID="2f58b078-73f0-4d86-a905-2a914d54687c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 13:07:58.212513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:58.211905 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 13:07:58.212513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:58.211988 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" podUID="ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 13:07:58.976675 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:58.976647 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 13:07:58.977140 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:58.977114 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" podUID="2f58b078-73f0-4d86-a905-2a914d54687c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 13:07:59.216513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:59.207876 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 13:07:59.216513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:59.207938 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" podUID="ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 13:07:59.449937 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:07:59.449901 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:07:59.977453 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:59.977421 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 13:07:59.977869 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:07:59.977472 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" podUID="2f58b078-73f0-4d86-a905-2a914d54687c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 13:08:00.209977 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:00.209944 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 13:08:00.210240 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:00.210204 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" podUID="ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 13:08:00.977699 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:00.977660 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 13:08:00.978085 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:00.977738 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" podUID="2f58b078-73f0-4d86-a905-2a914d54687c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 13:08:01.207498 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:01.207447 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 13:08:01.207674 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:01.207540 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" podUID="ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 13:08:02.052293 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:02.052257 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:08:02.053011 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:02.052988 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:08:02.054019 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:02.053994 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:08:02.559644 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:02.559612 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:08:02.560014 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:02.559987 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:08:02.560699 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:02.560679 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:08:23.098165 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.098123 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk"] Oct 30 13:08:23.102522 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.102498 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.136537 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.136005 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk"] Oct 30 13:08:23.196979 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.196928 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.197308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.197282 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.197414 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.197336 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.197549 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.197523 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.197616 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.197574 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cft8q\" (UniqueName: \"kubernetes.io/projected/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-kube-api-access-cft8q\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.197673 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.197657 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.197728 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.197714 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.197816 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.197760 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.197882 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.197829 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.298818 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.298781 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.299010 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.298836 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.299010 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.298877 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.299010 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.298919 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.299010 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.298991 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.299249 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.299022 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cft8q\" (UniqueName: \"kubernetes.io/projected/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-kube-api-access-cft8q\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.299249 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.299083 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.299249 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.299133 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.299249 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.299177 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.299445 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.299275 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.299445 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.299351 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.299714 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.299690 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.299899 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.299764 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.300101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.300084 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.301661 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.301638 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.301727 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.301674 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.324846 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.324811 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.324998 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.324961 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cft8q\" (UniqueName: \"kubernetes.io/projected/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-kube-api-access-cft8q\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.421598 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.421558 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:23.630539 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.630502 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2"] Oct 30 13:08:23.636279 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.636254 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.653893 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.653863 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2"] Oct 30 13:08:23.654349 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.654328 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ee7d65\"/\"distinct-w6-ee7d65-kserve-self-signed-certs\"" Oct 30 13:08:23.705502 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.705411 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-home\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.705502 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.705448 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-dshm\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.705763 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.705736 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-tmp-dir\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.705820 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.705801 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62bfs\" (UniqueName: \"kubernetes.io/projected/0d2f6484-3059-448a-976e-04230dc1f220-kube-api-access-62bfs\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.705861 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.705840 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-model-cache\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.705895 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.705870 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d2f6484-3059-448a-976e-04230dc1f220-tls-certs\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.766567 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.766538 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk"] Oct 30 13:08:23.773390 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.773348 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:23.773524 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.773442 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:23.773524 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.773507 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:23.806866 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.806831 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-home\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.807029 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.806879 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-dshm\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.807135 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.807031 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-tmp-dir\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.807135 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.807081 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62bfs\" (UniqueName: \"kubernetes.io/projected/0d2f6484-3059-448a-976e-04230dc1f220-kube-api-access-62bfs\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.807135 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.807114 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-model-cache\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.807286 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.807139 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d2f6484-3059-448a-976e-04230dc1f220-tls-certs\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.807545 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.807476 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-tmp-dir\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.807775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.807750 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-model-cache\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.807865 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.807791 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-home\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.809596 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.809544 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-dshm\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.810153 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.810129 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d2f6484-3059-448a-976e-04230dc1f220-tls-certs\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.847935 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.847247 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62bfs\" (UniqueName: \"kubernetes.io/projected/0d2f6484-3059-448a-976e-04230dc1f220-kube-api-access-62bfs\") pod \"distinct-w6-ee7d65-kserve-7c75b64879-5pxr2\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:23.954512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:23.953796 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:24.182580 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:24.182539 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" event={"ID":"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81","Type":"ContainerStarted","Data":"eb729006c3e68070423ca8715cd0e6998f17502f79ae4a324a07b20ade0287aa"} Oct 30 13:08:24.182984 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:24.182606 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" event={"ID":"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81","Type":"ContainerStarted","Data":"abd3c6aa98c4eb5c11a033c041e27aecd65562c6c26b9cc68fe3aebcd1f07c32"} Oct 30 13:08:24.236572 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:24.236453 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" podStartSLOduration=1.23643419 podStartE2EDuration="1.23643419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:24.235042175 +0000 UTC m=+960.867229338" watchObservedRunningTime="2026-10-30 13:08:24.23643419 +0000 UTC m=+960.868621352" Oct 30 13:08:24.321083 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:24.321025 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2"] Oct 30 13:08:24.322998 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:08:24.322958 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d2f6484_3059_448a_976e_04230dc1f220.slice/crio-6a8fcb097d335f2490fa06b8f0c942677f1f53778fe81f2bc78a035c7334048e WatchSource:0}: Error finding container 6a8fcb097d335f2490fa06b8f0c942677f1f53778fe81f2bc78a035c7334048e: Status 404 returned error can't find the container with id 6a8fcb097d335f2490fa06b8f0c942677f1f53778fe81f2bc78a035c7334048e Oct 30 13:08:24.422611 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:24.422577 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:24.424092 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:24.424068 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 13:08:24.424226 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:24.424111 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" podUID="22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 13:08:25.205514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:25.196018 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" event={"ID":"0d2f6484-3059-448a-976e-04230dc1f220","Type":"ContainerStarted","Data":"3ead712e4ee52220267b83d1cedfbe7f047a7ae1be6fd45624f6cdefcc885b06"} Oct 30 13:08:25.205514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:25.196062 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" event={"ID":"0d2f6484-3059-448a-976e-04230dc1f220","Type":"ContainerStarted","Data":"6a8fcb097d335f2490fa06b8f0c942677f1f53778fe81f2bc78a035c7334048e"} Oct 30 13:08:25.240100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:25.240048 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podStartSLOduration=2.240034194 podStartE2EDuration="2.240034194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:25.239176937 +0000 UTC m=+961.871364105" watchObservedRunningTime="2026-10-30 13:08:25.240034194 +0000 UTC m=+961.872221373" Oct 30 13:08:25.422213 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:25.422175 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 13:08:25.422516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:25.422462 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" podUID="22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 13:08:25.954307 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:25.954276 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:26.422099 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:26.422056 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 13:08:26.422564 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:26.422116 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" podUID="22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 13:08:27.449516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:27.433240 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 13:08:27.450175 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:27.450131 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" podUID="22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 13:08:28.428118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:28.428081 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 13:08:28.428456 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:28.428399 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" podUID="22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 13:08:28.529217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:28.528633 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:28.599552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:28.599478 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:28.660630 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:28.660579 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 13:08:28.785228 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:28.784597 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:28.922568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:28.922526 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 13:08:29.429294 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:29.428841 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 13:08:29.429294 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:29.428901 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" podUID="22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 13:08:29.432525 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:08:29.431961 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:08:30.422969 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:30.422929 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 13:08:30.423437 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:30.422998 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" podUID="22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 13:08:31.422603 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:31.422571 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 13:08:31.422766 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:31.422650 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" podUID="22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 13:08:33.319845 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:33.319814 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:33.320355 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:33.320138 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:33.321070 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:33.321049 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:08:33.955024 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:33.954988 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:33.967737 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:33.967708 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:34.290623 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:34.290550 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:37.764842 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.764804 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk"] Oct 30 13:08:37.769947 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.769929 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.793928 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.793901 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk"] Oct 30 13:08:37.878088 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.878052 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.878273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.878103 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.878273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.878247 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.878273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.878270 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.878387 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.878315 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.878387 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.878365 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4852\" (UniqueName: \"kubernetes.io/projected/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-kube-api-access-s4852\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.878560 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.878544 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.878601 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.878575 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.878636 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.878614 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.975746 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.975698 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg"] Oct 30 13:08:37.979744 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.979721 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:37.979981 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.979959 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.980070 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.980016 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.980127 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.980067 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.980180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.980161 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.980238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.980186 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.980349 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.980329 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.980437 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.980363 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4852\" (UniqueName: \"kubernetes.io/projected/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-kube-api-access-s4852\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.980437 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.980405 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.980566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.980502 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.980566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.980536 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.980676 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.980580 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.980676 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.980579 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.980676 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.980666 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.980916 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.980898 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.982663 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.982640 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:37.983082 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:37.983062 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:38.004041 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.004010 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4852\" (UniqueName: \"kubernetes.io/projected/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-kube-api-access-s4852\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:38.010798 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.010762 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg"] Oct 30 13:08:38.011559 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.011532 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:38.081947 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.081853 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239e461e-6255-4e2e-bc8a-b6c62c013c04-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.082087 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.082019 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.082087 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.082050 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.082087 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.082076 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.082252 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.082230 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6lp5\" (UniqueName: \"kubernetes.io/projected/239e461e-6255-4e2e-bc8a-b6c62c013c04-kube-api-access-s6lp5\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.082371 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.082352 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.082461 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.082440 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.082548 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.082483 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.082616 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.082581 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.083610 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.083592 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:38.140009 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.139881 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4"] Oct 30 13:08:38.145049 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.145027 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.168357 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.168322 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4"] Oct 30 13:08:38.184186 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.184157 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.184362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.184204 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.184362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.184267 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239e461e-6255-4e2e-bc8a-b6c62c013c04-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.184362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.184336 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.184362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.184359 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.184634 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.184393 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.184634 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.184437 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6lp5\" (UniqueName: \"kubernetes.io/projected/239e461e-6255-4e2e-bc8a-b6c62c013c04-kube-api-access-s6lp5\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.184634 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.184568 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.184784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.184649 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.185304 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.185205 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.185304 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.185260 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.185473 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.185439 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.185473 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.185461 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239e461e-6255-4e2e-bc8a-b6c62c013c04-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.185622 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.185528 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.187301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.187280 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.187436 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.187417 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.211720 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.211693 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.212140 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.212117 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6lp5\" (UniqueName: \"kubernetes.io/projected/239e461e-6255-4e2e-bc8a-b6c62c013c04-kube-api-access-s6lp5\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.286151 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.286113 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.286151 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.286152 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.286413 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.286295 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.286413 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.286359 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.286552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.286461 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.286619 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.286568 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.286619 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.286604 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snwwf\" (UniqueName: \"kubernetes.io/projected/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-kube-api-access-snwwf\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.286697 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.286682 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.286741 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.286715 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.308734 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.308702 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:38.388188 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.388148 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.388361 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.388208 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snwwf\" (UniqueName: \"kubernetes.io/projected/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-kube-api-access-snwwf\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.388361 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.388259 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.388361 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.388296 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.388558 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.388386 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.388558 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.388408 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.388558 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.388525 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.388745 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.388566 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.388745 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.388621 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.389160 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.389134 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istiod-ca-cert\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.389853 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.389827 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-workload-certs\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.389973 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.389917 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-workload-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.390054 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.390034 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-data\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.390145 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.390124 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-credential-socket\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.391899 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.391874 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-envoy\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.392172 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.392153 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-podinfo\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.414179 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.414154 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk"] Oct 30 13:08:38.420131 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.420086 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:38.420245 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.420179 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:38.420245 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.420237 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:38.421331 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.421309 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-token\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.421603 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.421581 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snwwf\" (UniqueName: \"kubernetes.io/projected/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-kube-api-access-snwwf\") pod \"e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.462698 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.462668 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:38.625638 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.625612 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg"] Oct 30 13:08:38.627987 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:08:38.627946 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239e461e_6255_4e2e_bc8a_b6c62c013c04.slice/crio-4211434e3c70ac173c7bfe13f92ed87c7ba53a6d0421495ad163548dc01eee6b WatchSource:0}: Error finding container 4211434e3c70ac173c7bfe13f92ed87c7ba53a6d0421495ad163548dc01eee6b: Status 404 returned error can't find the container with id 4211434e3c70ac173c7bfe13f92ed87c7ba53a6d0421495ad163548dc01eee6b Oct 30 13:08:38.631022 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.630988 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:38.631130 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.631060 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:38.631130 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.631093 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:38.724387 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.724355 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8"] Oct 30 13:08:38.729753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.729726 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:38.768577 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.767531 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8"] Oct 30 13:08:38.793819 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.793288 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4"] Oct 30 13:08:38.822727 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.821478 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:38.822727 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.821602 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:38.822727 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.821643 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:38.895514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.895449 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:38.895765 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.895709 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:38.895765 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.895750 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lgz9\" (UniqueName: \"kubernetes.io/projected/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-kube-api-access-4lgz9\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:38.895895 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.895787 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:38.895895 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.895822 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:38.895895 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.895847 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:38.896055 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.895910 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:38.896055 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.895947 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:38.896055 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:38.896003 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.002517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.001343 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.002517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.001528 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.002517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.001561 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lgz9\" (UniqueName: \"kubernetes.io/projected/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-kube-api-access-4lgz9\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.002517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.001599 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.002517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.001630 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.002517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.001653 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.002517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.001711 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.002517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.001750 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.002517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.001800 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.002517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.002268 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-data\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.004268 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.003973 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-workload-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.005986 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.004904 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-credential-socket\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.005986 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.005595 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istiod-ca-cert\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.006283 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.006237 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-workload-certs\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.010034 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.009987 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-podinfo\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.038111 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.038037 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-token\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.040821 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.040748 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lgz9\" (UniqueName: \"kubernetes.io/projected/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-kube-api-access-4lgz9\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.065181 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.063856 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-envoy\") pod \"e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.078533 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.078233 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:39.356999 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.356123 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" event={"ID":"6ea67e42-fdce-4f95-9b23-a62f6fab20dd","Type":"ContainerStarted","Data":"05c696ae8391022118f109b802d233fa8411ee26b30ad36afe0c5890f1846448"} Oct 30 13:08:39.356999 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.356199 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" event={"ID":"6ea67e42-fdce-4f95-9b23-a62f6fab20dd","Type":"ContainerStarted","Data":"36881eb748c9f411b1dd37b8fc6e9df1c21a5de1e11ee8881aefe29a1b4ea0b0"} Oct 30 13:08:39.359315 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.359272 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" event={"ID":"239e461e-6255-4e2e-bc8a-b6c62c013c04","Type":"ContainerStarted","Data":"b01c853ba4b158563b65f2a24374db2916364468d17630d3f8a516e286196e2a"} Oct 30 13:08:39.359810 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.359783 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" event={"ID":"239e461e-6255-4e2e-bc8a-b6c62c013c04","Type":"ContainerStarted","Data":"4211434e3c70ac173c7bfe13f92ed87c7ba53a6d0421495ad163548dc01eee6b"} Oct 30 13:08:39.363404 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.363373 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" event={"ID":"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9","Type":"ContainerStarted","Data":"67f3ef67bd346d3b3fd1f496ce7b388bc9e9e387ce5658a551b37c808558764d"} Oct 30 13:08:39.363584 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.363426 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" event={"ID":"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9","Type":"ContainerStarted","Data":"34ee4c6883686224b1377a05b75241fa052fae4dc5fe599d31bd9be0ccba7ba6"} Oct 30 13:08:39.464930 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.464841 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:39.466335 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.466303 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:08:39.466505 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.466364 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" podUID="6ea67e42-fdce-4f95-9b23-a62f6fab20dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 13:08:39.514242 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.514171 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" podStartSLOduration=1.5141528960000001 podStartE2EDuration="1.514152896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:39.42951308 +0000 UTC m=+976.061700231" watchObservedRunningTime="2026-10-30 13:08:39.514152896 +0000 UTC m=+976.146340059" Oct 30 13:08:39.536251 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.536180 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" podStartSLOduration=2.536155589 podStartE2EDuration="2.536155589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:39.513166335 +0000 UTC m=+976.145353515" watchObservedRunningTime="2026-10-30 13:08:39.536155589 +0000 UTC m=+976.168342751" Oct 30 13:08:39.552397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.546056 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8"] Oct 30 13:08:39.557803 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.554527 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:39.557803 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.554622 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:39.557803 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.554669 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:39.571608 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:39.571538 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" podStartSLOduration=2.571519694 podStartE2EDuration="2.571519694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:39.570758841 +0000 UTC m=+976.202946017" watchObservedRunningTime="2026-10-30 13:08:39.571519694 +0000 UTC m=+976.203706856" Oct 30 13:08:40.090584 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:40.089250 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:40.090584 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:40.089390 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 13:08:40.090584 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:40.089442 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" podUID="0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 13:08:40.322619 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:40.322120 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:40.322619 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:40.322254 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:08:40.322619 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:40.322301 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" podUID="239e461e-6255-4e2e-bc8a-b6c62c013c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 13:08:40.414515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:40.412140 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" event={"ID":"3bcbfc27-09bc-41ec-b7f7-8320b5e56172","Type":"ContainerStarted","Data":"4c770b2cd933f1a6bba22b9db989625dcdb33aa3696018576bb782c72d19bc47"} Oct 30 13:08:40.414515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:40.412216 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" event={"ID":"3bcbfc27-09bc-41ec-b7f7-8320b5e56172","Type":"ContainerStarted","Data":"c6403af4f3efe70d534aa9aa02a9cf78f79a3ffd144b6bb2d04e646ea6bf7acd"} Oct 30 13:08:40.486512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:40.459457 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" podStartSLOduration=2.459438952 podStartE2EDuration="2.459438952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:40.458043739 +0000 UTC m=+977.090230906" watchObservedRunningTime="2026-10-30 13:08:40.459438952 +0000 UTC m=+977.091626115" Oct 30 13:08:40.489477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:40.489445 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:08:40.490274 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:40.490242 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" podUID="6ea67e42-fdce-4f95-9b23-a62f6fab20dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 13:08:41.082563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.082516 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:41.082910 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.082889 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:08:41.084754 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.084724 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" podUID="3bcbfc27-09bc-41ec-b7f7-8320b5e56172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 13:08:41.085158 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.084662 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 13:08:41.085304 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.085284 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" podUID="0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 13:08:41.284878 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.284835 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:41.311953 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.311913 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:08:41.312234 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.312201 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" podUID="239e461e-6255-4e2e-bc8a-b6c62c013c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 13:08:41.465044 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.464985 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:08:41.465315 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.465281 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" podUID="6ea67e42-fdce-4f95-9b23-a62f6fab20dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 13:08:41.763381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.760592 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh"] Oct 30 13:08:41.844826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.844786 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:41.846269 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.846234 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh"] Oct 30 13:08:41.919526 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.914643 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-model-cache\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:41.926530 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.919925 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-home\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:41.926896 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.926863 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-dshm\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:41.927071 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.927053 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14c1b45-7310-4620-bc17-07354849809e-tls-certs\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:41.934514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.927158 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-tmp-dir\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:41.934904 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.934873 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vzkr\" (UniqueName: \"kubernetes.io/projected/a14c1b45-7310-4620-bc17-07354849809e-kube-api-access-8vzkr\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:41.989531 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:41.987304 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d833bd\"/\"distinct-w3-d833bd-kserve-self-signed-certs\"" Oct 30 13:08:42.038999 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.038658 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-model-cache\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:42.038999 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.038754 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-home\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:42.038999 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.038812 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-dshm\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:42.038999 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.038852 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14c1b45-7310-4620-bc17-07354849809e-tls-certs\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:42.038999 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.038877 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-tmp-dir\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:42.038999 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.038921 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vzkr\" (UniqueName: \"kubernetes.io/projected/a14c1b45-7310-4620-bc17-07354849809e-kube-api-access-8vzkr\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:42.084513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.042272 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-model-cache\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:42.084513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.042559 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-home\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:42.084513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.065085 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-tmp-dir\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:42.084513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.079308 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:08:42.084513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.079393 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" podUID="3bcbfc27-09bc-41ec-b7f7-8320b5e56172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 13:08:42.092668 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.091233 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14c1b45-7310-4620-bc17-07354849809e-tls-certs\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:42.103089 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.102119 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 13:08:42.103089 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.102184 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" podUID="0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 13:08:42.114530 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.104620 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vzkr\" (UniqueName: \"kubernetes.io/projected/a14c1b45-7310-4620-bc17-07354849809e-kube-api-access-8vzkr\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:42.135536 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.123334 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-dshm\") pod \"distinct-w3-d833bd-kserve-6f7b95885d-njwnh\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:42.167048 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.166815 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:42.286674 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.286384 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:42.287190 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.286972 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:42.329734 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.324050 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:08:42.329734 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.324112 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" podUID="239e461e-6255-4e2e-bc8a-b6c62c013c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 13:08:42.509521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.486190 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:42.510240 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.510148 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:08:42.510454 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.510425 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" podUID="6ea67e42-fdce-4f95-9b23-a62f6fab20dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 13:08:42.607570 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.603779 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:42.607570 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.604184 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": context deadline exceeded" Oct 30 13:08:42.661530 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.655683 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 13:08:42.669642 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.669591 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:42.811514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:42.786886 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:43.101693 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:43.091702 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:08:43.147466 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:43.147423 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" podUID="3bcbfc27-09bc-41ec-b7f7-8320b5e56172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 13:08:43.147716 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:43.096884 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 13:08:43.147894 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:43.147871 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" podUID="0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 13:08:43.300937 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:43.291803 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:43.579284 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:43.579237 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:08:43.580396 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:43.580142 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" podUID="6ea67e42-fdce-4f95-9b23-a62f6fab20dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 13:08:43.677338 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:43.677292 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:08:43.677630 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:43.677603 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" podUID="239e461e-6255-4e2e-bc8a-b6c62c013c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 13:08:43.681922 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:43.681881 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:43.721073 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:43.711561 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:44.130513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.126102 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 13:08:44.153730 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.152535 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 13:08:44.176993 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.176951 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:08:44.177246 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.177222 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" podUID="3bcbfc27-09bc-41ec-b7f7-8320b5e56172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 13:08:44.531363 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.502863 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 13:08:44.531363 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.505446 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 13:08:44.531363 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.529728 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 13:08:44.589553 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.570140 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:08:44.589553 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.570197 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" podUID="239e461e-6255-4e2e-bc8a-b6c62c013c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 13:08:44.740571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.719330 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:08:44.740571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.719406 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" podUID="6ea67e42-fdce-4f95-9b23-a62f6fab20dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 13:08:44.740571 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:08:44.723934 7010 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.202s" Oct 30 13:08:44.740571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.730853 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:08:44.740571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.730932 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:08:44.740571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.730967 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:08:44.740571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.731018 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:08:44.771530 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.765906 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:44.782657 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.782620 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 13:08:44.783007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.782956 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" podUID="0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 13:08:44.783231 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.783214 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh"] Oct 30 13:08:44.783983 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.783919 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:44.848282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:44.844402 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" event={"ID":"a14c1b45-7310-4620-bc17-07354849809e","Type":"ContainerStarted","Data":"420aab01ff73707574f09afdcf720745a156c09d575e691e6e863ba7ab7f8012"} Oct 30 13:08:45.094760 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.094636 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 13:08:45.095084 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.095050 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" podUID="0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 13:08:45.095585 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.095562 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:08:45.100617 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.100584 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" podUID="3bcbfc27-09bc-41ec-b7f7-8320b5e56172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 13:08:45.133687 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.130643 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.156924 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.156749 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.182701 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.182656 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:08:45.182998 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.182963 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 13:08:45.207775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.185839 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.207775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.186153 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:08:45.316523 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.279568 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.317419 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.317385 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:08:45.326519 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.317631 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" podUID="239e461e-6255-4e2e-bc8a-b6c62c013c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 13:08:45.326754 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.325703 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.493072 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.493034 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:08:45.493830 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.493278 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" podUID="6ea67e42-fdce-4f95-9b23-a62f6fab20dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 13:08:45.736740 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.724530 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.736740 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.730608 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.736740 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.731422 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.736740 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.732272 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.736740 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.732622 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:08:45.767596 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.767550 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.782514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.768133 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:08:45.785246 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.784337 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.893663 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.893612 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:45.991777 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:45.991669 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:46.211513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.205902 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 13:08:46.223441 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.213320 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 13:08:46.223441 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.215618 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:08:46.223441 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.215662 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" podUID="3bcbfc27-09bc-41ec-b7f7-8320b5e56172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 13:08:46.223441 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.222612 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 13:08:46.223441 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.222968 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:08:46.275567 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.214477 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 13:08:46.275567 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.248877 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" podUID="0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 13:08:46.368389 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.368358 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:08:46.633523 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.605043 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:08:46.654898 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.654862 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:08:46.655137 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.655101 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" podUID="6ea67e42-fdce-4f95-9b23-a62f6fab20dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 13:08:46.690535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.683594 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 13:08:46.690535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.683651 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" podUID="239e461e-6255-4e2e-bc8a-b6c62c013c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 13:08:46.703895 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.703849 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:08:46.732415 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.732378 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:08:46.734197 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.734171 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:08:46.746746 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.745307 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:08:46.768506 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:46.766329 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:47.061405 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:47.061216 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:08:47.069991 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:47.064302 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:47.104563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:47.089630 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:08:47.104563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:47.089701 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" podUID="3bcbfc27-09bc-41ec-b7f7-8320b5e56172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 13:08:47.149764 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:47.149731 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 13:08:47.150028 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:47.149998 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" podUID="0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 13:08:47.393519 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:47.388523 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" event={"ID":"a14c1b45-7310-4620-bc17-07354849809e","Type":"ContainerStarted","Data":"27d7d6d97303afbd413eb046a4d4f8625dcfdb1df030ec554d658771d9db3db8"} Oct 30 13:08:47.458824 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:47.458761 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" podStartSLOduration=6.458739766 podStartE2EDuration="6.458739766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:47.456554153 +0000 UTC m=+984.088741339" watchObservedRunningTime="2026-10-30 13:08:47.458739766 +0000 UTC m=+984.090926930" Oct 30 13:08:47.463359 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:47.463324 7010 patch_prober.go:29] interesting pod/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 13:08:47.463579 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:47.463382 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" podUID="6ea67e42-fdce-4f95-9b23-a62f6fab20dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 13:08:47.817845 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:47.817318 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:47.834930 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:47.832905 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:47.845932 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:47.845767 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:08:48.079567 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:48.078888 7010 patch_prober.go:29] interesting pod/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 13:08:48.079567 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:48.078960 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" podUID="3bcbfc27-09bc-41ec-b7f7-8320b5e56172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 13:08:48.095223 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:48.093384 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:48.095223 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:48.094187 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:48.095223 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:48.095047 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:08:48.167447 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:48.167412 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:48.798386 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:48.798354 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:48.800078 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:48.799728 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:48.801016 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:48.800987 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:08:49.237226 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:49.237190 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:49.237871 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:49.237803 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:49.239123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:49.238912 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:08:50.429458 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.429424 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn"] Oct 30 13:08:50.435620 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.435591 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.453283 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.453219 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn"] Oct 30 13:08:50.462524 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.462461 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-52a649\"/\"distinct-2-w1-52a649-kserve-self-signed-certs\"" Oct 30 13:08:50.514594 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.514550 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-home\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.514746 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.514610 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-model-cache\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.514854 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.514827 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e411b13-5f67-4954-bdcd-582026eb0ac4-tls-certs\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.515033 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.515003 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dbxm\" (UniqueName: \"kubernetes.io/projected/4e411b13-5f67-4954-bdcd-582026eb0ac4-kube-api-access-2dbxm\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.515199 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.515173 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-dshm\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.515444 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.515419 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-tmp-dir\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.617312 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.617267 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-home\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.617541 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.617327 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-model-cache\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.617541 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.617398 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e411b13-5f67-4954-bdcd-582026eb0ac4-tls-certs\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.617541 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.617469 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dbxm\" (UniqueName: \"kubernetes.io/projected/4e411b13-5f67-4954-bdcd-582026eb0ac4-kube-api-access-2dbxm\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.617717 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.617572 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-dshm\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.617717 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.617608 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-tmp-dir\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.618565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.617830 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-model-cache\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.618565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.618042 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-tmp-dir\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.619029 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.619004 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-home\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.624518 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.622509 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e411b13-5f67-4954-bdcd-582026eb0ac4-tls-certs\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.627519 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.626024 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-dshm\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.659274 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.659192 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dbxm\" (UniqueName: \"kubernetes.io/projected/4e411b13-5f67-4954-bdcd-582026eb0ac4-kube-api-access-2dbxm\") pod \"distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:50.750996 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:50.750898 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:51.081682 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:51.081594 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn"] Oct 30 13:08:51.086391 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:08:51.086363 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e411b13_5f67_4954_bdcd_582026eb0ac4.slice/crio-6c9e2a49636a73f670536bfc6a2b75996d8b65d62c40e869003942e605ea37e0 WatchSource:0}: Error finding container 6c9e2a49636a73f670536bfc6a2b75996d8b65d62c40e869003942e605ea37e0: Status 404 returned error can't find the container with id 6c9e2a49636a73f670536bfc6a2b75996d8b65d62c40e869003942e605ea37e0 Oct 30 13:08:51.419831 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:51.419790 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" event={"ID":"4e411b13-5f67-4954-bdcd-582026eb0ac4","Type":"ContainerStarted","Data":"0dc982ca5f8dd20192867ca34a120912fb46dd809dcc43fb9bae1177baf6f1e4"} Oct 30 13:08:51.419831 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:51.419834 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" event={"ID":"4e411b13-5f67-4954-bdcd-582026eb0ac4","Type":"ContainerStarted","Data":"6c9e2a49636a73f670536bfc6a2b75996d8b65d62c40e869003942e605ea37e0"} Oct 30 13:08:51.465313 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:51.465257 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" podStartSLOduration=1.465241738 podStartE2EDuration="1.465241738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:51.463195278 +0000 UTC m=+988.095382462" watchObservedRunningTime="2026-10-30 13:08:51.465241738 +0000 UTC m=+988.097428899" Oct 30 13:08:51.751947 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:51.751868 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:08:52.167375 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:52.167333 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:52.181232 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:52.181198 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:52.441840 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:52.441743 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:53.380198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.380163 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx"] Oct 30 13:08:53.384648 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.384626 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.404211 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.404186 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-cb58b8-openshift-default-dockercfg-dftld\"" Oct 30 13:08:53.420929 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.420882 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx"] Oct 30 13:08:53.465537 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.465478 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.465709 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.465609 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.465709 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.465654 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.465789 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.465712 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.465789 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.465757 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.465850 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.465804 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khlpc\" (UniqueName: \"kubernetes.io/projected/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-kube-api-access-khlpc\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.465850 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.465828 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.465937 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.465918 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.466015 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.465971 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.567885 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.567842 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.568148 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.568118 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.568303 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.568281 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.568542 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.568484 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.568767 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.568742 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.568767 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.568763 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.568943 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.568920 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.569028 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.569011 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.569142 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.569122 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.569203 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.569161 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.569260 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.569224 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khlpc\" (UniqueName: \"kubernetes.io/projected/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-kube-api-access-khlpc\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.569384 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.569124 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.569640 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.569618 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.569712 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.569618 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.571246 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.571222 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.571615 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.571596 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.598073 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.598033 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.598231 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.598150 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khlpc\" (UniqueName: \"kubernetes.io/projected/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-kube-api-access-khlpc\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:53.700044 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:53.699945 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:54.170064 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:08:54.170029 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1cd6b6_1ebf_41ad_a67d_2a0c30b9d6db.slice/crio-41f15b917aa37ee1243637ef80f4dfeb701bb7febb2b8277c8f8d83f5f64e384 WatchSource:0}: Error finding container 41f15b917aa37ee1243637ef80f4dfeb701bb7febb2b8277c8f8d83f5f64e384: Status 404 returned error can't find the container with id 41f15b917aa37ee1243637ef80f4dfeb701bb7febb2b8277c8f8d83f5f64e384 Oct 30 13:08:54.172375 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:54.172344 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx"] Oct 30 13:08:54.172813 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:54.172770 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:54.172917 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:54.172859 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:54.172917 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:54.172902 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:08:54.472573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:54.472244 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" event={"ID":"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db","Type":"ContainerStarted","Data":"5081caa1009928136a004e10a2777239f3c4ffc1f5d8e606b1a770d947f664c0"} Oct 30 13:08:54.472573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:54.472318 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" event={"ID":"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db","Type":"ContainerStarted","Data":"41f15b917aa37ee1243637ef80f4dfeb701bb7febb2b8277c8f8d83f5f64e384"} Oct 30 13:08:54.547483 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:54.547422 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" podStartSLOduration=1.547402452 podStartE2EDuration="1.547402452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:54.546958538 +0000 UTC m=+991.179145701" watchObservedRunningTime="2026-10-30 13:08:54.547402452 +0000 UTC m=+991.179589615" Oct 30 13:08:54.704513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:54.700635 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:08:55.615515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:55.614548 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:55.687868 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:55.687821 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:55.777822 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:55.777768 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:55.865576 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:55.863139 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:08:55.865576 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:55.863216 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" podUID="ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 13:08:55.865576 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:55.865261 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:55.944737 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:55.944678 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:56.096589 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:56.094638 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 13:08:56.523902 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:56.523855 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" podUID="a14c1b45-7310-4620-bc17-07354849809e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 13:08:56.662591 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:56.662541 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:56.663913 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:56.663885 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:08:56.677459 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:56.665251 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:56.725191 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:56.725137 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:56.743511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:56.732884 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:08:56.743511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:56.732937 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" podUID="ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 13:08:56.743511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:56.733030 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:08:56.755568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:56.753618 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:56.845768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:56.781254 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 13:08:56.845768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:56.808353 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:08:56.845768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:56.809423 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:56.845768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:56.842575 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:56.878668 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:56.878628 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:56.895390 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:56.895358 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:08:57.194322 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:57.194291 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:08:57.663374 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:57.663329 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:57.680661 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:57.680598 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:57.684512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:57.681300 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:08:57.765658 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:57.765613 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:57.773534 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:57.766597 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:08:57.774986 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:57.774952 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" podUID="ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 13:08:57.782625 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:57.782580 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:57.849076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:57.849035 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:57.849717 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:57.849693 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:08:57.864879 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:57.864832 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:57.865541 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:57.865518 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:08:57.932751 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:57.932651 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:57.938971 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:57.938938 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:08:58.438636 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:58.438594 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" podUID="a14c1b45-7310-4620-bc17-07354849809e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:58.731380 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:58.681326 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:58.731380 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:58.682515 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 13:08:58.731380 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:58.701737 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:08:58.731380 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:58.701800 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" podUID="ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 13:08:58.769060 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:58.768605 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:58.792521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:58.787616 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:58.853295 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:58.851130 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 13:08:58.886084 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:58.886040 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:58.945763 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:58.945719 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:59.095466 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:59.095334 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:59.294525 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:59.292313 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 13:08:59.448893 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:59.448847 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" podUID="a14c1b45-7310-4620-bc17-07354849809e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:59.450033 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:59.450009 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:08:59.685646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:59.685594 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:59.718267 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:59.718164 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:59.730755 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:59.730710 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:59.732303 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:59.732271 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:08:59.750797 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:59.750759 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" podUID="ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 13:08:59.820298 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:59.820246 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:59.825628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:59.825587 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:59.839640 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:08:59.822017 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:08:59.865808 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:59.865770 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:59.888641 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:59.888591 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:08:59.979699 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:08:59.979589 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:00.008514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:00.007829 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:09:00.148655 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:00.148608 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:00.163579 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:00.162752 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:09:00.277973 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:00.277891 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:09:00.295446 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:00.290270 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:09:00.295446 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:00.291757 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:09:00.295446 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:00.292354 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:09:00.312774 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:00.312299 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:09:00.324315 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:00.314040 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:09:00.333202 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:00.331961 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:09:00.348449 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:00.348424 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:09:00.732562 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:00.720966 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:09:00.732562 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:00.721027 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" podUID="ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 13:09:00.751800 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:00.751768 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:09:00.768586 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:00.768555 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:09:01.523687 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:01.523657 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:09:01.718452 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:01.718411 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:09:01.718747 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:01.718478 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" podUID="ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 13:09:02.705976 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:02.705938 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:09:02.706599 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:02.706567 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" podUID="ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 13:09:03.704514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:03.702921 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 13:09:03.704514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:03.702986 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" podUID="ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 13:09:05.086968 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:05.086925 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:09:05.088594 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:05.088571 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:09:05.089395 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:05.089372 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:09:22.534668 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:22.534616 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" podUID="4e411b13-5f67-4954-bdcd-582026eb0ac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:23.184647 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.175703 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl"] Oct 30 13:09:23.184647 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.181757 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.307528 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.294253 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl"] Oct 30 13:09:23.328732 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.328696 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.328974 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.328950 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.329294 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.329273 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.329520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.329474 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.339169 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.335619 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.339169 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.335659 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crpwz\" (UniqueName: \"kubernetes.io/projected/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-kube-api-access-crpwz\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.339169 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.335716 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.339169 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.335763 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.339169 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.335846 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.439272 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.439180 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.439533 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.439510 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.439783 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.439759 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.439932 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.439915 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.440063 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.440048 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.440177 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.440159 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crpwz\" (UniqueName: \"kubernetes.io/projected/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-kube-api-access-crpwz\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.440315 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.440300 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.440426 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.440411 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.440575 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.440558 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.441276 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.441252 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-data\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.441704 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.441674 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-workload-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.445560 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.445537 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-credential-socket\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.446473 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.446455 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istiod-ca-cert\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.447674 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.447643 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-workload-certs\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.526092 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.484184 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-envoy\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.526092 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.525210 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-podinfo\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.550516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.534901 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crpwz\" (UniqueName: \"kubernetes.io/projected/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-kube-api-access-crpwz\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.554502 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.554454 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-token\") pod \"e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.769951 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.745076 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:23.995397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:23.995357 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7"] Oct 30 13:09:24.008505 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.007428 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.020124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.020027 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7"] Oct 30 13:09:24.104259 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.100283 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.104622 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.104583 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.104789 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.104768 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2rjq\" (UniqueName: \"kubernetes.io/projected/355db35c-0262-415b-8df0-bede943852c7-kube-api-access-c2rjq\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.104968 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.104950 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/355db35c-0262-415b-8df0-bede943852c7-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.105098 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.105081 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.105206 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.105191 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/355db35c-0262-415b-8df0-bede943852c7-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.105441 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.105424 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.105848 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.105827 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.106066 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.106049 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/355db35c-0262-415b-8df0-bede943852c7-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.107139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.106680 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn"] Oct 30 13:09:24.133777 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.133745 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.138660 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.138269 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn"] Oct 30 13:09:24.208597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.207797 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.208597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.207872 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3891d8b5-1466-46b3-a720-de605147a7ba-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.208597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.207930 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.208597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.207963 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2rjq\" (UniqueName: \"kubernetes.io/projected/355db35c-0262-415b-8df0-bede943852c7-kube-api-access-c2rjq\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.208597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.208020 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.208597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.208058 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.208597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.208086 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.208597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.208138 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.208597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.208167 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.208597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.208247 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.208597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.208296 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/355db35c-0262-415b-8df0-bede943852c7-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.208597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.208351 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s687m\" (UniqueName: \"kubernetes.io/projected/3891d8b5-1466-46b3-a720-de605147a7ba-kube-api-access-s687m\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.208597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.208396 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3891d8b5-1466-46b3-a720-de605147a7ba-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.208597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.208468 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/355db35c-0262-415b-8df0-bede943852c7-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.208597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.208528 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.208597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.208572 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/355db35c-0262-415b-8df0-bede943852c7-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.208597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.208603 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3891d8b5-1466-46b3-a720-de605147a7ba-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.209610 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.208652 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.213173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.211692 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-istio-envoy\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.213173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.212136 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-istio-data\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.213173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.212426 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/355db35c-0262-415b-8df0-bede943852c7-istiod-ca-cert\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.213173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.212783 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-credential-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.213173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.212947 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-workload-socket\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.213173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.213092 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-workload-certs\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.216196 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.216168 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/355db35c-0262-415b-8df0-bede943852c7-istio-podinfo\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.237535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.236810 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/355db35c-0262-415b-8df0-bede943852c7-istio-token\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.237535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.237348 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2rjq\" (UniqueName: \"kubernetes.io/projected/355db35c-0262-415b-8df0-bede943852c7-kube-api-access-c2rjq\") pod \"e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.310244 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.310150 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3891d8b5-1466-46b3-a720-de605147a7ba-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.310420 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.310243 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.310420 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.310266 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.310420 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.310376 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s687m\" (UniqueName: \"kubernetes.io/projected/3891d8b5-1466-46b3-a720-de605147a7ba-kube-api-access-s687m\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.310615 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.310418 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3891d8b5-1466-46b3-a720-de605147a7ba-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.310615 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.310473 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.310615 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.310528 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3891d8b5-1466-46b3-a720-de605147a7ba-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.310615 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.310571 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.310871 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.310655 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.310871 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.310710 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.310871 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.310730 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.311021 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.310990 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3891d8b5-1466-46b3-a720-de605147a7ba-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.311058 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.311043 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.311091 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.311075 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.312707 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.312684 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.312815 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.312798 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3891d8b5-1466-46b3-a720-de605147a7ba-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.334962 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.334932 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3891d8b5-1466-46b3-a720-de605147a7ba-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.335263 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.335240 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s687m\" (UniqueName: \"kubernetes.io/projected/3891d8b5-1466-46b3-a720-de605147a7ba-kube-api-access-s687m\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.346584 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.346556 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl"] Oct 30 13:09:24.349509 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:09:24.348088 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b1816f_01c8_4c93_8ba3_ffdd5a1be50d.slice/crio-f0302edd47f695d02c872331432b80180e5ca4eb48842beb014982cf5e52af05 WatchSource:0}: Error finding container f0302edd47f695d02c872331432b80180e5ca4eb48842beb014982cf5e52af05: Status 404 returned error can't find the container with id f0302edd47f695d02c872331432b80180e5ca4eb48842beb014982cf5e52af05 Oct 30 13:09:24.350408 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.350383 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:24.352339 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.352304 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:24.352415 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.352388 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:24.352450 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.352431 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:24.468241 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.467911 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:24.700552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.700516 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7"] Oct 30 13:09:24.702763 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:09:24.702737 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355db35c_0262_415b_8df0_bede943852c7.slice/crio-f8e9c766cc71edaed7e8f6059eb663dd8a789450a905c7ab5120dd56ff4475d3 WatchSource:0}: Error finding container f8e9c766cc71edaed7e8f6059eb663dd8a789450a905c7ab5120dd56ff4475d3: Status 404 returned error can't find the container with id f8e9c766cc71edaed7e8f6059eb663dd8a789450a905c7ab5120dd56ff4475d3 Oct 30 13:09:24.705920 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.705889 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:24.706007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.705952 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:24.706007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.705980 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:24.811850 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.811786 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn"] Oct 30 13:09:24.812800 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:09:24.812772 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3891d8b5_1466_46b3_a720_de605147a7ba.slice/crio-64bedbcef8b6bfe65a4609bfe89cb915c8ab8ecbecb3c7482d58228b2bfef0c5 WatchSource:0}: Error finding container 64bedbcef8b6bfe65a4609bfe89cb915c8ab8ecbecb3c7482d58228b2bfef0c5: Status 404 returned error can't find the container with id 64bedbcef8b6bfe65a4609bfe89cb915c8ab8ecbecb3c7482d58228b2bfef0c5 Oct 30 13:09:24.816539 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.816482 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:24.816629 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.816590 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:24.816668 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.816655 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:24.909217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.909178 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" event={"ID":"3891d8b5-1466-46b3-a720-de605147a7ba","Type":"ContainerStarted","Data":"64bedbcef8b6bfe65a4609bfe89cb915c8ab8ecbecb3c7482d58228b2bfef0c5"} Oct 30 13:09:24.911322 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.911288 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" event={"ID":"355db35c-0262-415b-8df0-bede943852c7","Type":"ContainerStarted","Data":"16751400b853e0df61780310f1d8c68d39bb65bf594b1ad73d0c1708cedbd106"} Oct 30 13:09:24.911430 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.911351 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" event={"ID":"355db35c-0262-415b-8df0-bede943852c7","Type":"ContainerStarted","Data":"f8e9c766cc71edaed7e8f6059eb663dd8a789450a905c7ab5120dd56ff4475d3"} Oct 30 13:09:24.913703 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.913628 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" event={"ID":"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d","Type":"ContainerStarted","Data":"9af61fdb229f6152c7f3200bded503340a66729535da56c7702ace8ed3f19a3e"} Oct 30 13:09:24.913703 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.913673 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" event={"ID":"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d","Type":"ContainerStarted","Data":"f0302edd47f695d02c872331432b80180e5ca4eb48842beb014982cf5e52af05"} Oct 30 13:09:24.974688 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:24.974623 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" podStartSLOduration=1.974601587 podStartE2EDuration="1.974601587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:24.964753819 +0000 UTC m=+1021.596940984" watchObservedRunningTime="2026-10-30 13:09:24.974601587 +0000 UTC m=+1021.606788748" Oct 30 13:09:25.021393 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:25.021330 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" podStartSLOduration=2.021309691 podStartE2EDuration="2.021309691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:25.013272125 +0000 UTC m=+1021.645459288" watchObservedRunningTime="2026-10-30 13:09:25.021309691 +0000 UTC m=+1021.653496857" Oct 30 13:09:25.353048 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:25.352972 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:25.354445 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:25.354405 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 13:09:25.354576 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:25.354535 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" podUID="355db35c-0262-415b-8df0-bede943852c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 13:09:25.753037 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:25.752943 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:25.764516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:25.762067 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:09:25.764516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:25.762139 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" podUID="03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:09:25.961710 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:25.961675 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" event={"ID":"3891d8b5-1466-46b3-a720-de605147a7ba","Type":"ContainerStarted","Data":"12b19b00f36697d838e7394c10c26c0701660f82558cb806600dd40d6cf27ddc"} Oct 30 13:09:26.025481 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:26.024689 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" podStartSLOduration=2.02466379 podStartE2EDuration="2.02466379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:26.01650633 +0000 UTC m=+1022.648693490" watchObservedRunningTime="2026-10-30 13:09:26.02466379 +0000 UTC m=+1022.656850953" Oct 30 13:09:26.369578 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:26.364107 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 13:09:26.369578 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:26.364163 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" podUID="355db35c-0262-415b-8df0-bede943852c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 13:09:26.470623 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:26.470531 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:26.472826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:26.472389 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 13:09:26.472826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:26.472454 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" podUID="3891d8b5-1466-46b3-a720-de605147a7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 13:09:26.749000 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:26.748897 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:09:26.749253 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:26.749224 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" podUID="03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:09:27.352935 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:27.352888 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 13:09:27.353543 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:27.353508 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" podUID="355db35c-0262-415b-8df0-bede943852c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 13:09:27.468673 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:27.468636 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 13:09:27.468855 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:27.468701 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" podUID="3891d8b5-1466-46b3-a720-de605147a7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 13:09:27.752512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:27.751918 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:09:27.752512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:27.751988 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" podUID="03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:09:28.359384 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:28.358898 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 13:09:28.359384 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:28.358977 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" podUID="355db35c-0262-415b-8df0-bede943852c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 13:09:28.469043 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:28.469008 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 13:09:28.469312 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:28.469282 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" podUID="3891d8b5-1466-46b3-a720-de605147a7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 13:09:28.797704 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:28.797665 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:09:28.799523 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:28.799468 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" podUID="03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:09:29.354847 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:29.354791 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 13:09:29.355167 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:29.355136 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" podUID="355db35c-0262-415b-8df0-bede943852c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 13:09:29.472090 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:29.472004 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 13:09:29.472609 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:29.472091 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" podUID="3891d8b5-1466-46b3-a720-de605147a7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 13:09:29.475373 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:09:29.475329 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:09:29.766517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:29.765670 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:09:29.766517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:29.765734 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" podUID="03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:09:30.359518 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:30.355624 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 13:09:30.368687 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:30.368641 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" podUID="355db35c-0262-415b-8df0-bede943852c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 13:09:30.469574 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:30.469538 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 13:09:30.469747 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:30.469602 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" podUID="3891d8b5-1466-46b3-a720-de605147a7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 13:09:30.754521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:30.753681 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:09:30.754521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:30.753741 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" podUID="03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:09:31.351509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:31.351400 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 13:09:31.351509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:31.351458 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" podUID="355db35c-0262-415b-8df0-bede943852c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 13:09:31.468685 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:31.468640 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 13:09:31.468867 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:31.468713 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" podUID="3891d8b5-1466-46b3-a720-de605147a7ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 13:09:31.749816 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:31.749767 7010 patch_prober.go:29] interesting pod/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 13:09:31.750039 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:31.749845 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" podUID="03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 13:09:32.351517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:32.351462 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 13:09:32.352096 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:32.351547 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" podUID="355db35c-0262-415b-8df0-bede943852c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 13:09:32.572655 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:32.572621 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:32.574524 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:32.574480 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:32.575762 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:32.575742 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:09:32.754475 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:32.754434 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:32.756155 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:32.756125 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:32.756661 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:32.756638 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:09:33.356285 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:33.356252 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:33.356824 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:33.356680 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:33.357878 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:33.357855 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:09:36.044333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.044290 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt"] Oct 30 13:09:36.050653 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.050621 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.066285 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.066257 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt"] Oct 30 13:09:36.075688 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.075658 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ee7d65\"/\"distinct-2-w6-ee7d65-kserve-self-signed-certs\"" Oct 30 13:09:36.214662 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.214621 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-model-cache\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.215004 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.214979 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxx2f\" (UniqueName: \"kubernetes.io/projected/0895771d-085b-4642-a9ea-3a7549d99831-kube-api-access-nxx2f\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.215169 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.215153 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0895771d-085b-4642-a9ea-3a7549d99831-tls-certs\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.215316 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.215300 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-dshm\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.215513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.215478 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-home\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.215776 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.215757 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-tmp-dir\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.317016 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.316916 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-dshm\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.317214 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.317020 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-home\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.317214 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.317139 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-tmp-dir\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.317331 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.317233 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-model-cache\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.317331 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.317316 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxx2f\" (UniqueName: \"kubernetes.io/projected/0895771d-085b-4642-a9ea-3a7549d99831-kube-api-access-nxx2f\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.317444 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.317363 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0895771d-085b-4642-a9ea-3a7549d99831-tls-certs\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.318418 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.318386 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-tmp-dir\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.325065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.322916 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-model-cache\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.331509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.326876 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-home\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.353516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.342987 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-dshm\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.353516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.348787 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0895771d-085b-4642-a9ea-3a7549d99831-tls-certs\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.358246 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.356705 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxx2f\" (UniqueName: \"kubernetes.io/projected/0895771d-085b-4642-a9ea-3a7549d99831-kube-api-access-nxx2f\") pod \"distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.372746 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.372423 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:36.696891 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:36.696858 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt"] Oct 30 13:09:36.698918 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:09:36.698886 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0895771d_085b_4642_a9ea_3a7549d99831.slice/crio-47428acf6484639c2d5744b40b5c36abc76ab446282dd126112a3e981a12a7a6 WatchSource:0}: Error finding container 47428acf6484639c2d5744b40b5c36abc76ab446282dd126112a3e981a12a7a6: Status 404 returned error can't find the container with id 47428acf6484639c2d5744b40b5c36abc76ab446282dd126112a3e981a12a7a6 Oct 30 13:09:37.352339 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:37.352301 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" event={"ID":"0895771d-085b-4642-a9ea-3a7549d99831","Type":"ContainerStarted","Data":"c87ed39f29a0fb23e601c021b917ac80226faa71829731478055df89825adb3e"} Oct 30 13:09:37.352339 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:37.352349 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" event={"ID":"0895771d-085b-4642-a9ea-3a7549d99831","Type":"ContainerStarted","Data":"47428acf6484639c2d5744b40b5c36abc76ab446282dd126112a3e981a12a7a6"} Oct 30 13:09:37.372809 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:37.372779 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:37.429414 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:37.429358 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" podStartSLOduration=1.429342401 podStartE2EDuration="1.429342401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:37.427392397 +0000 UTC m=+1034.059579559" watchObservedRunningTime="2026-10-30 13:09:37.429342401 +0000 UTC m=+1034.061529560" Oct 30 13:09:46.373123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:46.373085 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:46.386730 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:46.386697 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:46.436110 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:46.436083 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:50.196513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:50.195579 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:50.300746 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:50.300679 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:50.324287 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:50.324244 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" podUID="a14c1b45-7310-4620-bc17-07354849809e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:50.332585 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:50.331836 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:50.427512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:50.426577 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" podUID="0895771d-085b-4642-a9ea-3a7549d99831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:50.481578 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:50.480657 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" podUID="4e411b13-5f67-4954-bdcd-582026eb0ac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:50.690646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:50.690599 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:50.691237 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:50.690762 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:50.768691 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:50.768580 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:50.796904 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:50.796858 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 13:09:50.855558 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:50.854779 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:50.895515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:50.891593 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:51.215518 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.215204 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 13:09:51.251512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.217638 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:09:51.316516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.315749 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:51.341522 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.317184 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:09:51.341522 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.339561 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" podUID="a14c1b45-7310-4620-bc17-07354849809e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:51.341522 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.339885 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:09:51.427009 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.403575 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:51.427009 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.408184 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:09:51.451515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.445118 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" podUID="0895771d-085b-4642-a9ea-3a7549d99831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:51.451515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.446192 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:51.521587 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.509112 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" podUID="4e411b13-5f67-4954-bdcd-582026eb0ac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 13:09:51.534513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.530546 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:09:51.691511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.691446 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:51.734609 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.693254 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:51.758632 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.758585 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" podUID="4e411b13-5f67-4954-bdcd-582026eb0ac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 30 13:09:51.773247 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.773121 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:51.801362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.801308 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:51.939307 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.939256 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:09:51.943780 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.943718 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:09:51.956197 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:51.956154 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:09:52.005633 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.005595 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:09:52.197213 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.196558 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:09:52.231520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.217171 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:09:52.264508 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.240454 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:09:52.296245 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.295985 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:52.302869 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.302833 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:09:52.303431 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.303396 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:52.340049 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.339990 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:52.443544 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.408741 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 13:09:52.443544 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.434416 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:52.443544 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.436009 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:09:52.540570 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.521860 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" podUID="4e411b13-5f67-4954-bdcd-582026eb0ac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 13:09:52.555325 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.555282 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": context deadline exceeded" Oct 30 13:09:52.638514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.581750 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:09:52.709520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.708661 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:52.720698 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.720638 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 13:09:52.859568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.857433 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:52.964664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.960735 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:09:52.964664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:52.961214 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:09:53.041063 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.041026 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54"] Oct 30 13:09:53.180834 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.180784 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54"] Oct 30 13:09:53.181280 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.181186 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.200758 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.200640 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:53.316305 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.314577 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 13:09:53.389542 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.387982 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 13:09:53.418537 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.418481 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:53.462873 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.462773 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqjzb\" (UniqueName: \"kubernetes.io/projected/000e5172-d652-4668-b6b7-cb8ee3f14ad9-kube-api-access-bqjzb\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.482627 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.482588 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.482933 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.482918 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.483117 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.483102 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.502789 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.502748 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.503170 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.503154 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.503275 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.503262 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.503382 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.503368 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.503468 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.503456 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.522594 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.522548 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" podUID="4e411b13-5f67-4954-bdcd-582026eb0ac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:53.543093 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.543061 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:09:53.572566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.572538 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:09:53.607313 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.605626 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqjzb\" (UniqueName: \"kubernetes.io/projected/000e5172-d652-4668-b6b7-cb8ee3f14ad9-kube-api-access-bqjzb\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.607313 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.605695 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.607313 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.605773 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.607313 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.605864 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.607313 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.605919 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.607313 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.606098 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.607313 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.606134 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.607313 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.606181 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.607313 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.606211 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.609788 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.609760 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-credential-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.610650 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.610622 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istiod-ca-cert\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.611345 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.611322 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-data\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.617846 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.611732 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-workload-socket\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.618045 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.617777 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-workload-certs\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.726064 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.725775 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqjzb\" (UniqueName: \"kubernetes.io/projected/000e5172-d652-4668-b6b7-cb8ee3f14ad9-kube-api-access-bqjzb\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.831512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.786106 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-token\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:53.889521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.861211 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:53.889521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.861680 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:09:53.970943 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:53.970122 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-podinfo\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:54.252736 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:54.252698 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:09:54.472504 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:54.468647 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:54.472504 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:54.469201 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:09:54.539932 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:54.529963 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:54.569733 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:54.569664 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-envoy\") pod \"e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:54.694056 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:54.694022 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:09:54.745939 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:54.745882 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:09:54.777324 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:54.777290 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:54.831911 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:54.831875 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:09:54.864136 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:54.864086 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 13:09:54.893921 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:54.893893 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:09:55.195520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:55.195473 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54"] Oct 30 13:09:55.202314 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:55.202283 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:55.202403 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:55.202352 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:55.202403 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:55.202384 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:09:55.270667 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:55.270628 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" event={"ID":"000e5172-d652-4668-b6b7-cb8ee3f14ad9","Type":"ContainerStarted","Data":"a9d341b610f58f5c3208d080edbf7fc36b40ebdf16e31a87bdf21a89e856bf5a"} Oct 30 13:09:56.282435 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:56.282377 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" event={"ID":"000e5172-d652-4668-b6b7-cb8ee3f14ad9","Type":"ContainerStarted","Data":"32aa77ef006c852f3813c2383ec0f531cbe5823145fde38d8f8d9959ad815d33"} Oct 30 13:09:56.343281 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:56.343227 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" podStartSLOduration=3.343213517 podStartE2EDuration="3.343213517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:56.339964437 +0000 UTC m=+1052.972151599" watchObservedRunningTime="2026-10-30 13:09:56.343213517 +0000 UTC m=+1052.975400678" Oct 30 13:09:56.778419 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:56.778388 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:09:56.779649 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:56.779628 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 13:09:56.779713 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:56.779673 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" podUID="000e5172-d652-4668-b6b7-cb8ee3f14ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 13:09:57.779342 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:57.779309 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 13:09:57.779927 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:57.779897 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" podUID="000e5172-d652-4668-b6b7-cb8ee3f14ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 13:09:58.778773 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:58.778731 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 13:09:58.778952 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:58.778790 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" podUID="000e5172-d652-4668-b6b7-cb8ee3f14ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 13:09:59.430738 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:09:59.430701 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:09:59.779184 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:59.779091 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 13:09:59.779184 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:09:59.779163 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" podUID="000e5172-d652-4668-b6b7-cb8ee3f14ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 13:10:00.778814 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:00.778774 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 13:10:00.779268 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:00.778833 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" podUID="000e5172-d652-4668-b6b7-cb8ee3f14ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 13:10:01.778347 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:01.778305 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 13:10:01.778577 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:01.778370 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" podUID="000e5172-d652-4668-b6b7-cb8ee3f14ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 13:10:02.778774 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:02.778733 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 13:10:02.779202 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:02.778805 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" podUID="000e5172-d652-4668-b6b7-cb8ee3f14ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 13:10:03.783338 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:03.783300 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:10:03.783765 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:03.783679 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:10:03.784532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:03.784511 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:10:16.921653 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:16.921617 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t"] Oct 30 13:10:16.925849 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:16.925826 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:16.944029 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:16.943997 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 13:10:16.944029 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:16.944004 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 13:10:16.944286 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:16.944085 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-glws9\"" Oct 30 13:10:16.956477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:16.956442 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 13:10:16.961656 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:16.961617 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t"] Oct 30 13:10:17.022272 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.022221 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sxjx\" (UniqueName: \"kubernetes.io/projected/e3b65b07-570d-49f3-8b74-c613b6e8c669-kube-api-access-5sxjx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.022477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.022344 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e3b65b07-570d-49f3-8b74-c613b6e8c669-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.022477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.022408 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e3b65b07-570d-49f3-8b74-c613b6e8c669-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.022477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.022460 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b65b07-570d-49f3-8b74-c613b6e8c669-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.022716 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.022694 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3b65b07-570d-49f3-8b74-c613b6e8c669-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.022817 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.022797 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b65b07-570d-49f3-8b74-c613b6e8c669-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.022882 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.022867 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e3b65b07-570d-49f3-8b74-c613b6e8c669-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.123851 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.123817 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b65b07-570d-49f3-8b74-c613b6e8c669-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.124063 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.123930 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3b65b07-570d-49f3-8b74-c613b6e8c669-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.124063 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.123975 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b65b07-570d-49f3-8b74-c613b6e8c669-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.124063 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.124010 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e3b65b07-570d-49f3-8b74-c613b6e8c669-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.124231 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.124105 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sxjx\" (UniqueName: \"kubernetes.io/projected/e3b65b07-570d-49f3-8b74-c613b6e8c669-kube-api-access-5sxjx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.124231 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.124176 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e3b65b07-570d-49f3-8b74-c613b6e8c669-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.124231 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.124216 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e3b65b07-570d-49f3-8b74-c613b6e8c669-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.124960 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.124932 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e3b65b07-570d-49f3-8b74-c613b6e8c669-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.126424 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.126399 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e3b65b07-570d-49f3-8b74-c613b6e8c669-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.126529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.126456 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b65b07-570d-49f3-8b74-c613b6e8c669-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.126782 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.126766 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e3b65b07-570d-49f3-8b74-c613b6e8c669-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.126879 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.126859 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3b65b07-570d-49f3-8b74-c613b6e8c669-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.167144 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.167110 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b65b07-570d-49f3-8b74-c613b6e8c669-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.168483 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.168453 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sxjx\" (UniqueName: \"kubernetes.io/projected/e3b65b07-570d-49f3-8b74-c613b6e8c669-kube-api-access-5sxjx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8nb8t\" (UID: \"e3b65b07-570d-49f3-8b74-c613b6e8c669\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.245441 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.238752 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:17.621649 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.621602 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t"] Oct 30 13:10:17.638553 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:17.638527 7010 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:10:18.470716 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:18.470674 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" event={"ID":"e3b65b07-570d-49f3-8b74-c613b6e8c669","Type":"ContainerStarted","Data":"fe81a68497344c12b06a9747bb532ff972e3de07a2c14c5c05b589b0fb47c4bd"} Oct 30 13:10:21.962514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:21.961683 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:22.102515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:22.101592 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:22.568516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:22.561620 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": context deadline exceeded" Oct 30 13:10:22.568516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:22.562123 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 13:10:22.587516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:22.579853 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 13:10:22.946526 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:22.942291 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:22.946526 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:22.942391 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:23.012856 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:22.984113 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:23.012856 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:22.985984 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:10:23.107512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:23.086642 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:23.205809 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:23.193932 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" podUID="a14c1b45-7310-4620-bc17-07354849809e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:23.268800 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:23.223721 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:23.268800 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:23.224284 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" podUID="0895771d-085b-4642-a9ea-3a7549d99831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:23.268800 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:23.224354 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 13:10:23.372570 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:23.369248 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 13:10:23.555515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:23.526149 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" podUID="4e411b13-5f67-4954-bdcd-582026eb0ac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 13:10:23.752896 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:23.619030 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:23.752896 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:23.648695 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 13:10:23.752896 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:23.326438 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" podUID="a14c1b45-7310-4620-bc17-07354849809e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 13:10:23.987647 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:23.987589 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:24.020518 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.014427 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:24.114751 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.114623 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:24.167644 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.167580 7010 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8nb8t container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:10:24.182680 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.182631 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" podUID="e3b65b07-570d-49f3-8b74-c613b6e8c669" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:10:24.182977 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.175208 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7"] Oct 30 13:10:24.195350 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.195312 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" event={"ID":"e3b65b07-570d-49f3-8b74-c613b6e8c669","Type":"ContainerStarted","Data":"d9d8da6a7429dfd153cc016e4b2f7b8aa2b82c2fb630c72c33cfcdb070c2fbfa"} Oct 30 13:10:24.215537 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.210901 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:24.216018 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.215854 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:10:24.216226 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.216212 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:10:24.218325 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.218307 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:10:24.218705 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.218688 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:10:24.218936 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.218920 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7"] Oct 30 13:10:24.219785 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.219768 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:24.391723 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.376621 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 13:10:24.391723 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.377176 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:10:24.391723 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.389075 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" podStartSLOduration=3.087467593 podStartE2EDuration="8.389054782s" totalImagesPullingTime="5.301587189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:24.36069561 +0000 UTC m=+1080.992882773" watchObservedRunningTime="2026-10-30 13:10:24.389054782 +0000 UTC m=+1081.021241945" Oct 30 13:10:24.403256 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.403210 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:24.453363 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.444252 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" podUID="0895771d-085b-4642-a9ea-3a7549d99831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 13:10:24.495266 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.468165 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:10:24.780478 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.771393 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:24.780478 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.771478 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:24.780478 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.771694 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:24.780478 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.771771 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:24.780478 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.771876 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88kq6\" (UniqueName: \"kubernetes.io/projected/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-kube-api-access-88kq6\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:24.780478 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.771930 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:24.780478 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.772012 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:24.780478 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.772050 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:24.780478 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.772170 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:24.781713 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.781173 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" podUID="a14c1b45-7310-4620-bc17-07354849809e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:24.781713 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.781598 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:10:24.798235 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.782538 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:24.798235 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.782863 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:10:25.001170 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.899460 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:25.001170 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.899560 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:25.001170 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.899695 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:25.001170 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.924591 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:25.001170 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.924694 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88kq6\" (UniqueName: \"kubernetes.io/projected/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-kube-api-access-88kq6\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:25.001170 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.924739 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:25.001170 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.924827 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:25.001170 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.924876 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:25.001170 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.924952 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:25.001170 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.927212 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:25.001170 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.927520 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:25.001170 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:24.968990 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:10:25.008199 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.008161 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:25.027612 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.027551 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:25.046908 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.046807 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:25.047895 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.047863 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:25.066332 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.066288 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:25.086503 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.086438 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:25.206966 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.206921 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:25.232664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.232613 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 13:10:25.321588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.321269 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:10:25.439126 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.437715 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88kq6\" (UniqueName: \"kubernetes.io/projected/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-kube-api-access-88kq6\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:25.516522 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.474935 7010 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8nb8t container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:10:25.516522 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.474991 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" podUID="e3b65b07-570d-49f3-8b74-c613b6e8c669" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:10:25.516522 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.477363 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:25.522523 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.522346 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:10:25.667728 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.642762 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" podUID="4e411b13-5f67-4954-bdcd-582026eb0ac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:25.815554 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.815155 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 30 13:10:25.815759 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.815555 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" podUID="a14c1b45-7310-4620-bc17-07354849809e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 13:10:25.851500 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.851446 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:25.885921 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.852223 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:26.011773 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.859513 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:26.012722 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.875890 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:26.013829 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.993609 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:26.014006 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:25.993664 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 13:10:26.026614 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:26.011580 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 13:10:26.066921 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:26.041064 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:26.115194 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:26.027068 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:26.118168 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:26.106602 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:26.236365 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:26.235127 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:26.286785 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:26.264172 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:26.373923 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:26.373037 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" podUID="0895771d-085b-4642-a9ea-3a7549d99831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:26.376527 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:26.375025 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:10:26.403630 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:26.403586 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:10:26.411760 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:26.407162 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:10:26.445629 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:26.445602 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:10:26.613574 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:26.605075 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:26.835838 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:26.835787 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" podUID="a14c1b45-7310-4620-bc17-07354849809e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 13:10:27.057890 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:27.057845 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:10:27.172082 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:27.159410 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:10:27.284722 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:27.284643 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8nb8t" Oct 30 13:10:27.634193 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:27.634161 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7"] Oct 30 13:10:27.634816 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:10:27.634784 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3da8c6a_cc2f_4c0d_9b5a_d04027578b4d.slice/crio-c66ded16f623bd72de2239914dbe733675f165121d7bb5f4b4dfb7093ecc8fd7 WatchSource:0}: Error finding container c66ded16f623bd72de2239914dbe733675f165121d7bb5f4b4dfb7093ecc8fd7: Status 404 returned error can't find the container with id c66ded16f623bd72de2239914dbe733675f165121d7bb5f4b4dfb7093ecc8fd7 Oct 30 13:10:27.637924 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:27.637888 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:27.638011 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:27.637977 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:27.638050 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:27.638022 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:28.230096 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:28.230053 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" event={"ID":"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d","Type":"ContainerStarted","Data":"115ca13f1368588b7fe939059b58eae2947469d984b19c219ee1f7af523fec25"} Oct 30 13:10:28.230598 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:28.230121 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" event={"ID":"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d","Type":"ContainerStarted","Data":"c66ded16f623bd72de2239914dbe733675f165121d7bb5f4b4dfb7093ecc8fd7"} Oct 30 13:10:28.293911 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:28.292542 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" podStartSLOduration=5.292517958 podStartE2EDuration="5.292517958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:28.288510533 +0000 UTC m=+1084.920697723" watchObservedRunningTime="2026-10-30 13:10:28.292517958 +0000 UTC m=+1084.924705133" Oct 30 13:10:28.670681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:28.670642 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:28.672036 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:28.672010 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 13:10:28.672141 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:28.672057 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" podUID="e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 13:10:29.424740 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:10:29.424703 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:10:29.670762 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:29.670726 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 13:10:29.670934 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:29.670782 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" podUID="e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 13:10:30.670196 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:30.670159 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 13:10:30.670706 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:30.670216 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" podUID="e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 13:10:31.669940 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:31.669906 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 13:10:31.670144 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:31.669966 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" podUID="e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 13:10:32.670886 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:32.670852 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 13:10:32.671301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:32.670905 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" podUID="e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 13:10:33.670546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:33.670482 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 13:10:33.670716 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:33.670567 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" podUID="e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 13:10:34.670684 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:34.670653 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 13:10:34.671079 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:34.670706 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" podUID="e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 13:10:35.674883 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:35.674851 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:35.675358 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:35.675331 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:35.676197 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:35.676175 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:10:38.769415 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:38.769382 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29"] Oct 30 13:10:38.774157 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:38.774135 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:38.806765 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:38.806734 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29"] Oct 30 13:10:38.930443 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:38.930405 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:38.930643 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:38.930552 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:38.930643 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:38.930610 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:38.930722 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:38.930713 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:38.930755 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:38.930748 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2gl5\" (UniqueName: \"kubernetes.io/projected/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-kube-api-access-s2gl5\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:38.930825 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:38.930813 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:38.930860 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:38.930842 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:38.931017 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:38.930995 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:38.931053 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:38.931030 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.032848 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.032746 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.032848 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.032839 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.033141 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.032886 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2gl5\" (UniqueName: \"kubernetes.io/projected/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-kube-api-access-s2gl5\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.033141 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.032955 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.033141 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.032990 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.033141 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.033091 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.033141 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.033124 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.033397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.033170 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.033397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.033226 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.033397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.033244 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.033397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.033317 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.033397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.033347 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.033397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.033389 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.033909 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.033884 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.035596 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.035572 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.035758 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.035739 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.057012 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.056982 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.057226 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.057205 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2gl5\" (UniqueName: \"kubernetes.io/projected/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-kube-api-access-s2gl5\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.088772 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.088738 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:39.416788 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.416762 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29"] Oct 30 13:10:39.417509 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:10:39.417461 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5dacfc7_1c21_4ac5_b93a_1f0763af424f.slice/crio-76e9682e213da1bd7f0eae0189b593674430b78d6d40be5e75b310bfabad722e WatchSource:0}: Error finding container 76e9682e213da1bd7f0eae0189b593674430b78d6d40be5e75b310bfabad722e: Status 404 returned error can't find the container with id 76e9682e213da1bd7f0eae0189b593674430b78d6d40be5e75b310bfabad722e Oct 30 13:10:39.420296 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.420269 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:39.420362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.420332 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:39.420403 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:39.420360 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:40.324717 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:40.324682 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" event={"ID":"d5dacfc7-1c21-4ac5-b93a-1f0763af424f","Type":"ContainerStarted","Data":"b6ad17091fa82f57b2459a93cc957fd3579b189e7e811bb685ee4a4496d026e0"} Oct 30 13:10:40.325362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:40.325335 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" event={"ID":"d5dacfc7-1c21-4ac5-b93a-1f0763af424f","Type":"ContainerStarted","Data":"76e9682e213da1bd7f0eae0189b593674430b78d6d40be5e75b310bfabad722e"} Oct 30 13:10:41.104576 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:41.104537 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:41.104938 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:41.104913 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 13:10:41.105105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:41.105080 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" podUID="d5dacfc7-1c21-4ac5-b93a-1f0763af424f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 13:10:42.109514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:42.100586 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 13:10:42.109514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:42.100649 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" podUID="d5dacfc7-1c21-4ac5-b93a-1f0763af424f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 13:10:43.094917 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:43.094878 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 13:10:43.095241 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:43.095211 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" podUID="d5dacfc7-1c21-4ac5-b93a-1f0763af424f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 13:10:44.093514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:44.092465 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 13:10:44.093514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:44.092653 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" podUID="d5dacfc7-1c21-4ac5-b93a-1f0763af424f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 13:10:45.109060 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:45.109014 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 13:10:45.109610 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:45.109101 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" podUID="d5dacfc7-1c21-4ac5-b93a-1f0763af424f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 13:10:46.089216 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:46.089174 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 13:10:46.089420 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:46.089225 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" podUID="d5dacfc7-1c21-4ac5-b93a-1f0763af424f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 13:10:47.093403 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:47.093369 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:47.093845 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:47.093811 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:47.094617 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:47.094596 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:10:47.157614 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:47.157553 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" podStartSLOduration=9.157534351 podStartE2EDuration="9.157534351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:40.407759626 +0000 UTC m=+1097.039946791" watchObservedRunningTime="2026-10-30 13:10:47.157534351 +0000 UTC m=+1103.789721513" Oct 30 13:10:54.108190 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.108150 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb"] Oct 30 13:10:54.357513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.357015 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb"] Oct 30 13:10:54.357513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.357131 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.435469 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.435425 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.439582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.439547 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.468531 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.439837 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/102a82c8-155f-4927-baa7-ae53a3b0fee1-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.575609 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.575473 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.577581 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.576443 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.578257 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.578235 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.578807 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.578784 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dq8k\" (UniqueName: \"kubernetes.io/projected/102a82c8-155f-4927-baa7-ae53a3b0fee1-kube-api-access-7dq8k\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.598892 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.579137 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.599427 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.599406 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.709568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.708642 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.709568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.708694 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.709568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.708753 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dq8k\" (UniqueName: \"kubernetes.io/projected/102a82c8-155f-4927-baa7-ae53a3b0fee1-kube-api-access-7dq8k\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.709568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.708824 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.709568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.708885 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.709568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.708972 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.709568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.708996 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.709568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.709080 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/102a82c8-155f-4927-baa7-ae53a3b0fee1-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.709568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.709184 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.710154 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.710041 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.729456 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.726950 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.729456 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.727217 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.729456 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.727890 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/102a82c8-155f-4927-baa7-ae53a3b0fee1-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.730161 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.730136 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.782533 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.774250 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.795849 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.789950 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.795849 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.790396 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.795849 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.792592 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dq8k\" (UniqueName: \"kubernetes.io/projected/102a82c8-155f-4927-baa7-ae53a3b0fee1-kube-api-access-7dq8k\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.827535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.824367 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" podUID="a14c1b45-7310-4620-bc17-07354849809e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:54.845066 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.843581 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:54.867345 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.866613 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:54.887640 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.886630 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:54.910033 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.909761 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:54.981286 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:54.981247 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 13:10:55.010581 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:55.009596 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 13:10:55.044606 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:55.041777 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:55.044606 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:55.042579 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:55.256570 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:55.256248 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:55.257058 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:55.256622 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": context deadline exceeded" Oct 30 13:10:55.323513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:55.322586 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" podUID="0895771d-085b-4642-a9ea-3a7549d99831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:55.409512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:55.406741 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:55.677958 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:55.663599 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:55.690447 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:55.690085 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" podUID="4e411b13-5f67-4954-bdcd-582026eb0ac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 13:10:55.919542 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:55.897596 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:55.919542 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:55.899939 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:10:55.919542 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:55.912130 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:10:55.993951 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:55.993907 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb"] Oct 30 13:10:56.002375 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:56.001153 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:10:56.050948 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:56.032415 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:56.050948 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:56.032524 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:56.050948 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:56.032572 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:10:56.880804 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:56.880765 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" event={"ID":"102a82c8-155f-4927-baa7-ae53a3b0fee1","Type":"ContainerStarted","Data":"5d579517abb1dd817fa70828bca205c4c388a3e2f404d2d63d85c4e1709051f0"} Oct 30 13:10:56.881185 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:56.880827 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" event={"ID":"102a82c8-155f-4927-baa7-ae53a3b0fee1","Type":"ContainerStarted","Data":"1b08e839e9cf2f4fc137e81fca97f13e51ae3a4d88c55a1fe6d5685387c157be"} Oct 30 13:10:56.914890 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:56.914844 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:10:56.963460 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:56.963403 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" podStartSLOduration=2.9633878989999998 podStartE2EDuration="2.963387899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:56.947466111 +0000 UTC m=+1113.579653302" watchObservedRunningTime="2026-10-30 13:10:56.963387899 +0000 UTC m=+1113.595575061" Oct 30 13:10:57.917056 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:57.917022 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:10:57.917435 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:57.917074 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" podUID="102a82c8-155f-4927-baa7-ae53a3b0fee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 13:10:58.914922 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:58.914881 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:10:58.915116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:58.914934 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" podUID="102a82c8-155f-4927-baa7-ae53a3b0fee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 13:10:59.426806 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:10:59.426769 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:10:59.915637 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:59.915602 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:10:59.915869 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:10:59.915678 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" podUID="102a82c8-155f-4927-baa7-ae53a3b0fee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 13:11:00.915840 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:00.915809 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:11:00.916260 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:00.915872 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" podUID="102a82c8-155f-4927-baa7-ae53a3b0fee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 13:11:01.917661 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:01.915883 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:11:01.917661 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:01.915942 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" podUID="102a82c8-155f-4927-baa7-ae53a3b0fee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 13:11:02.915784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:02.915751 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 13:11:02.916010 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:02.915819 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" podUID="102a82c8-155f-4927-baa7-ae53a3b0fee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 13:11:03.980139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:03.980100 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:11:03.981619 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:03.981591 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:11:03.982857 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:03.982831 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:11:19.581776 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:19.580574 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:19.737348 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:19.737302 7010 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:19.831927 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:19.831823 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" podUID="a14c1b45-7310-4620-bc17-07354849809e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:19.879302 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:19.879257 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:19.983517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:19.982822 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:20.001223 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:20.001181 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:20.017614 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:20.017568 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:20.019620 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:20.019590 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:20.020756 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:20.020730 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:20.319542 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:11:20.314342 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" podUID="0895771d-085b-4642-a9ea-3a7549d99831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:11:29.402245 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:11:29.402201 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:11:59.410664 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:11:59.410589 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:12:14.442957 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.442918 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk"] Oct 30 13:12:14.450053 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.450026 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.490726 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.490684 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3778dacd-openshift-default-dockercfg-drb2j\"" Oct 30 13:12:14.494189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.494159 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk"] Oct 30 13:12:14.523697 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.523663 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-credential-socket\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.523966 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.523946 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/428e36fe-a829-4bee-bcd5-5150d5358411-istio-podinfo\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.524064 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.523982 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-workload-socket\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.524125 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.524096 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/428e36fe-a829-4bee-bcd5-5150d5358411-istiod-ca-cert\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.524202 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.524182 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/428e36fe-a829-4bee-bcd5-5150d5358411-istio-token\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.524261 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.524228 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-istio-envoy\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.524328 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.524308 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbnmj\" (UniqueName: \"kubernetes.io/projected/428e36fe-a829-4bee-bcd5-5150d5358411-kube-api-access-xbnmj\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.524386 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.524355 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-workload-certs\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.524442 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.524402 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-istio-data\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.627681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.627638 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/428e36fe-a829-4bee-bcd5-5150d5358411-istio-podinfo\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.643511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.632553 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-workload-socket\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.643828 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.643793 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/428e36fe-a829-4bee-bcd5-5150d5358411-istiod-ca-cert\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.644019 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.644004 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/428e36fe-a829-4bee-bcd5-5150d5358411-istio-token\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.644135 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.644122 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-istio-envoy\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.644329 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.644317 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbnmj\" (UniqueName: \"kubernetes.io/projected/428e36fe-a829-4bee-bcd5-5150d5358411-kube-api-access-xbnmj\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.644448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.644437 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-workload-certs\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.644773 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.632244 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/428e36fe-a829-4bee-bcd5-5150d5358411-istio-podinfo\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.654548 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.645014 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-workload-socket\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.654548 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.645499 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/428e36fe-a829-4bee-bcd5-5150d5358411-istiod-ca-cert\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.683514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.664423 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-workload-certs\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.683769 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.664566 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-istio-data\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.684147 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.684125 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-credential-socket\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.684402 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.667844 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-istio-envoy\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.684527 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.683157 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbnmj\" (UniqueName: \"kubernetes.io/projected/428e36fe-a829-4bee-bcd5-5150d5358411-kube-api-access-xbnmj\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.684642 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.664818 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-istio-data\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.692543 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.685161 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-credential-socket\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.728561 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.704543 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/428e36fe-a829-4bee-bcd5-5150d5358411-istio-token\") pod \"e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:14.780973 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:14.780931 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:15.203783 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:15.203746 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk"] Oct 30 13:12:15.219914 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:15.219867 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:15.220190 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:15.220023 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:15.220190 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:15.220092 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:16.046884 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:16.046849 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" event={"ID":"428e36fe-a829-4bee-bcd5-5150d5358411","Type":"ContainerStarted","Data":"7360cc0b840e330e07e05af9d2af3e4e62d4efc51841d2b2034a81ca71be709f"} Oct 30 13:12:16.047278 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:16.046931 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" event={"ID":"428e36fe-a829-4bee-bcd5-5150d5358411","Type":"ContainerStarted","Data":"729f86345c78928af8cf83fbba445f5e51eb194d9e5e9fce7742e301b13b7f08"} Oct 30 13:12:16.118608 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:16.118544 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" podStartSLOduration=2.118523387 podStartE2EDuration="2.118523387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:16.114370324 +0000 UTC m=+1192.746557486" watchObservedRunningTime="2026-10-30 13:12:16.118523387 +0000 UTC m=+1192.750710549" Oct 30 13:12:16.782237 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:16.782162 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:16.798401 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:16.798368 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:17.063969 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:17.063895 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:17.065261 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:17.065233 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:12:23.649453 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:23.649422 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:12:23.652948 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:23.652925 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:12:29.422845 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:29.422786 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:12:40.865761 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:40.865724 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz"] Oct 30 13:12:40.870197 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:40.870176 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:12:40.889630 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:40.889604 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz"] Oct 30 13:12:40.898539 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:40.898509 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3778dacd-serving-cert\"" Oct 30 13:12:40.955898 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:40.955851 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-maas-api-tls\") pod \"maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:12:40.956070 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:40.955969 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-openshift-service-ca\") pod \"maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:12:40.956116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:40.956084 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f52v7\" (UniqueName: \"kubernetes.io/projected/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-kube-api-access-f52v7\") pod \"maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:12:40.956162 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:40.956118 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-metrics-certs\") pod \"maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:12:41.069609 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:41.069553 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-metrics-certs\") pod \"maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:12:41.069828 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:41.069818 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-maas-api-tls\") pod \"maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:12:41.069901 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:41.069876 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-openshift-service-ca\") pod \"maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:12:41.070005 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:41.069968 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f52v7\" (UniqueName: \"kubernetes.io/projected/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-kube-api-access-f52v7\") pod \"maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:12:41.070172 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:41.070106 7010 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-serving-cert: secret "maas-api-e2e-mt-3778dacd-serving-cert" not found Oct 30 13:12:41.070236 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:41.070208 7010 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-maas-api-tls podName:5d4b02d9-cc6d-4783-8c2f-c9c39215e18c nodeName:}" failed. No retries permitted until 2026-10-30 13:12:41.570183679 +0000 UTC m=+1218.202370854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-maas-api-tls") pod "maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" (UID: "5d4b02d9-cc6d-4783-8c2f-c9c39215e18c") : secret "maas-api-e2e-mt-3778dacd-serving-cert" not found Oct 30 13:12:41.074621 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:41.074588 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-openshift-service-ca\") pod \"maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:12:41.090821 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:41.090738 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-metrics-certs\") pod \"maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:12:41.105241 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:41.105204 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f52v7\" (UniqueName: \"kubernetes.io/projected/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-kube-api-access-f52v7\") pod \"maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:12:41.576030 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:41.575988 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-maas-api-tls\") pod \"maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:12:41.578766 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:41.578736 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-maas-api-tls\") pod \"maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:12:41.792509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:41.792241 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:12:42.167366 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:42.167332 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz"] Oct 30 13:12:42.168580 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:12:42.168553 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4b02d9_cc6d_4783_8c2f_c9c39215e18c.slice/crio-6e0fe9699646b9388f6989923513bd84df3817af91b4485077b9f119acf9cde2 WatchSource:0}: Error finding container 6e0fe9699646b9388f6989923513bd84df3817af91b4485077b9f119acf9cde2: Status 404 returned error can't find the container with id 6e0fe9699646b9388f6989923513bd84df3817af91b4485077b9f119acf9cde2 Oct 30 13:12:42.270051 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:42.270006 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" event={"ID":"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c","Type":"ContainerStarted","Data":"6e0fe9699646b9388f6989923513bd84df3817af91b4485077b9f119acf9cde2"} Oct 30 13:12:43.278701 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:43.278666 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" event={"ID":"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c","Type":"ContainerStarted","Data":"c5995f589c0a666b8dc9dcd79862719add86f32442768980518654bb5c6ba77e"} Oct 30 13:12:43.279089 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:43.278794 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:12:43.325112 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:43.325063 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" podStartSLOduration=3.038829184 podStartE2EDuration="3.32504935s" totalImagesPullingTime="286.220166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:43.323781437 +0000 UTC m=+1219.955968621" watchObservedRunningTime="2026-10-30 13:12:43.32504935 +0000 UTC m=+1219.957236511" Oct 30 13:12:43.694280 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:43.694249 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8746-263ede"] Oct 30 13:12:43.698564 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:43.698545 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8746-263ede" Oct 30 13:12:43.715985 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:43.715957 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8746-263ede"] Oct 30 13:12:43.765373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:43.765348 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nqkjx\"" Oct 30 13:12:43.823193 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:43.823163 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r27fx\" (UniqueName: \"kubernetes.io/projected/b9027bd5-2cde-4261-bd56-6054887c345f-kube-api-access-r27fx\") pod \"test-curl-8746-263ede\" (UID: \"b9027bd5-2cde-4261-bd56-6054887c345f\") " pod="opendatahub/test-curl-8746-263ede" Oct 30 13:12:43.924802 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:43.924771 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r27fx\" (UniqueName: \"kubernetes.io/projected/b9027bd5-2cde-4261-bd56-6054887c345f-kube-api-access-r27fx\") pod \"test-curl-8746-263ede\" (UID: \"b9027bd5-2cde-4261-bd56-6054887c345f\") " pod="opendatahub/test-curl-8746-263ede" Oct 30 13:12:43.948746 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:43.948690 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r27fx\" (UniqueName: \"kubernetes.io/projected/b9027bd5-2cde-4261-bd56-6054887c345f-kube-api-access-r27fx\") pod \"test-curl-8746-263ede\" (UID: \"b9027bd5-2cde-4261-bd56-6054887c345f\") " pod="opendatahub/test-curl-8746-263ede" Oct 30 13:12:44.011109 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:44.011076 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8746-263ede" Oct 30 13:12:44.317961 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:44.317929 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8746-263ede"] Oct 30 13:12:44.319644 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:12:44.319600 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9027bd5_2cde_4261_bd56_6054887c345f.slice/crio-c0bef2ea312d751d9682743a8d85cd4d01d7853d4fb4f680067cae4ba4c38ce8 WatchSource:0}: Error finding container c0bef2ea312d751d9682743a8d85cd4d01d7853d4fb4f680067cae4ba4c38ce8: Status 404 returned error can't find the container with id c0bef2ea312d751d9682743a8d85cd4d01d7853d4fb4f680067cae4ba4c38ce8 Oct 30 13:12:45.298962 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:45.298908 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8746-263ede" event={"ID":"b9027bd5-2cde-4261-bd56-6054887c345f","Type":"ContainerStarted","Data":"c0bef2ea312d751d9682743a8d85cd4d01d7853d4fb4f680067cae4ba4c38ce8"} Oct 30 13:12:47.317198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:47.317148 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8746-263ede" event={"ID":"b9027bd5-2cde-4261-bd56-6054887c345f","Type":"ContainerStarted","Data":"ab649230cc57d5c9eaac2ee86f2ec00fd7ebf4b491b68070b5edd887da31d3cb"} Oct 30 13:12:47.358083 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:47.358030 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8746-263ede" podStartSLOduration=2.156342826 podStartE2EDuration="4.358016127s" totalImagesPullingTime="2.201673301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:47.356526135 +0000 UTC m=+1223.988713297" watchObservedRunningTime="2026-10-30 13:12:47.358016127 +0000 UTC m=+1223.990203288" Oct 30 13:12:48.359708 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:48.359671 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8746-263ede"] Oct 30 13:12:48.362917 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:48.362882 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8746-263ede"] Oct 30 13:12:48.516246 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:48.516210 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8746-b1fb9f"] Oct 30 13:12:48.516912 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:48.516898 7010 cpu_manager.go:414] "containerMap: removing container" podUID="b9027bd5-2cde-4261-bd56-6054887c345f" containerName="test-curl-8746-263ede" Oct 30 13:12:48.516979 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:48.516914 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9027bd5-2cde-4261-bd56-6054887c345f" containerName="test-curl-8746-263ede" Oct 30 13:12:48.517030 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:48.517021 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9027bd5-2cde-4261-bd56-6054887c345f" containerName="test-curl-8746-263ede" Oct 30 13:12:48.520605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:48.520589 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8746-b1fb9f" Oct 30 13:12:48.541253 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:48.541223 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8746-b1fb9f"] Oct 30 13:12:48.608125 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:48.608089 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djkmd\" (UniqueName: \"kubernetes.io/projected/7f64527f-ec69-45a9-b9a3-99255f6e007e-kube-api-access-djkmd\") pod \"test-curl-8746-b1fb9f\" (UID: \"7f64527f-ec69-45a9-b9a3-99255f6e007e\") " pod="opendatahub/test-curl-8746-b1fb9f" Oct 30 13:12:48.710316 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:48.710274 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djkmd\" (UniqueName: \"kubernetes.io/projected/7f64527f-ec69-45a9-b9a3-99255f6e007e-kube-api-access-djkmd\") pod \"test-curl-8746-b1fb9f\" (UID: \"7f64527f-ec69-45a9-b9a3-99255f6e007e\") " pod="opendatahub/test-curl-8746-b1fb9f" Oct 30 13:12:48.735007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:48.734974 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djkmd\" (UniqueName: \"kubernetes.io/projected/7f64527f-ec69-45a9-b9a3-99255f6e007e-kube-api-access-djkmd\") pod \"test-curl-8746-b1fb9f\" (UID: \"7f64527f-ec69-45a9-b9a3-99255f6e007e\") " pod="opendatahub/test-curl-8746-b1fb9f" Oct 30 13:12:48.835314 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:48.832812 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8746-b1fb9f" Oct 30 13:12:49.170835 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:49.170808 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8746-b1fb9f"] Oct 30 13:12:49.172465 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:12:49.172427 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f64527f_ec69_45a9_b9a3_99255f6e007e.slice/crio-b7b270fe9e1fefe18dcbab2d91b312db636cdb5d7e2088445580f084d440ff40 WatchSource:0}: Error finding container b7b270fe9e1fefe18dcbab2d91b312db636cdb5d7e2088445580f084d440ff40: Status 404 returned error can't find the container with id b7b270fe9e1fefe18dcbab2d91b312db636cdb5d7e2088445580f084d440ff40 Oct 30 13:12:49.291035 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:49.291012 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:12:49.333022 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:49.332985 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8746-b1fb9f" event={"ID":"7f64527f-ec69-45a9-b9a3-99255f6e007e","Type":"ContainerStarted","Data":"316aab0dce90e66dcbd46d02434f423069eac2e551eae2fa24b416839697a1d7"} Oct 30 13:12:49.333182 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:49.333031 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8746-b1fb9f" event={"ID":"7f64527f-ec69-45a9-b9a3-99255f6e007e","Type":"ContainerStarted","Data":"b7b270fe9e1fefe18dcbab2d91b312db636cdb5d7e2088445580f084d440ff40"} Oct 30 13:12:49.333182 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:49.333088 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8746-263ede" podUID="b9027bd5-2cde-4261-bd56-6054887c345f" containerName="test-curl-8746-263ede" containerID="cri-o://ab649230cc57d5c9eaac2ee86f2ec00fd7ebf4b491b68070b5edd887da31d3cb" gracePeriod=2 Oct 30 13:12:49.348731 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:49.348699 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-263ede\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="b9027bd5-2cde-4261-bd56-6054887c345f" pod="opendatahub/test-curl-8746-263ede" Oct 30 13:12:49.392416 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:49.392349 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8746-b1fb9f" podStartSLOduration=1.392329729 podStartE2EDuration="1.392329729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:49.388813856 +0000 UTC m=+1226.021001019" watchObservedRunningTime="2026-10-30 13:12:49.392329729 +0000 UTC m=+1226.024516893" Oct 30 13:12:50.306352 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:50.306303 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8746-b1fb9f"] Oct 30 13:12:50.311284 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:50.311254 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8746-b1fb9f"] Oct 30 13:12:50.460427 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:50.460390 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8746-063bf8"] Oct 30 13:12:50.461222 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:50.461200 7010 cpu_manager.go:414] "containerMap: removing container" podUID="7f64527f-ec69-45a9-b9a3-99255f6e007e" containerName="test-curl-8746-b1fb9f" Oct 30 13:12:50.461326 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:50.461224 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f64527f-ec69-45a9-b9a3-99255f6e007e" containerName="test-curl-8746-b1fb9f" Oct 30 13:12:50.461431 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:50.461417 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f64527f-ec69-45a9-b9a3-99255f6e007e" containerName="test-curl-8746-b1fb9f" Oct 30 13:12:50.466189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:50.466160 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8746-063bf8" Oct 30 13:12:50.479420 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:50.479391 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8746-063bf8"] Oct 30 13:12:50.539655 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:50.539621 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8d7v\" (UniqueName: \"kubernetes.io/projected/deb9418e-f302-40db-bb61-264dab5d8875-kube-api-access-l8d7v\") pod \"test-curl-8746-063bf8\" (UID: \"deb9418e-f302-40db-bb61-264dab5d8875\") " pod="opendatahub/test-curl-8746-063bf8" Oct 30 13:12:50.641533 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:50.641483 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8d7v\" (UniqueName: \"kubernetes.io/projected/deb9418e-f302-40db-bb61-264dab5d8875-kube-api-access-l8d7v\") pod \"test-curl-8746-063bf8\" (UID: \"deb9418e-f302-40db-bb61-264dab5d8875\") " pod="opendatahub/test-curl-8746-063bf8" Oct 30 13:12:50.670229 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:50.670195 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8d7v\" (UniqueName: \"kubernetes.io/projected/deb9418e-f302-40db-bb61-264dab5d8875-kube-api-access-l8d7v\") pod \"test-curl-8746-063bf8\" (UID: \"deb9418e-f302-40db-bb61-264dab5d8875\") " pod="opendatahub/test-curl-8746-063bf8" Oct 30 13:12:50.787301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:50.787264 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8746-063bf8" Oct 30 13:12:51.097438 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:51.097406 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8746-063bf8"] Oct 30 13:12:51.098794 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:12:51.098764 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddeb9418e_f302_40db_bb61_264dab5d8875.slice/crio-69d7fa839276f3e2043accb2fedf96947c18e1c9238f9132e14aeb7104424fb7 WatchSource:0}: Error finding container 69d7fa839276f3e2043accb2fedf96947c18e1c9238f9132e14aeb7104424fb7: Status 404 returned error can't find the container with id 69d7fa839276f3e2043accb2fedf96947c18e1c9238f9132e14aeb7104424fb7 Oct 30 13:12:51.350679 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:51.350604 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8746-063bf8" event={"ID":"deb9418e-f302-40db-bb61-264dab5d8875","Type":"ContainerStarted","Data":"6b920ddfc22cd40b9cd5e7c5e8d7575411a1e41e1f2e8e8c3e6b46402ba91f4a"} Oct 30 13:12:51.350679 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:51.350655 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8746-063bf8" event={"ID":"deb9418e-f302-40db-bb61-264dab5d8875","Type":"ContainerStarted","Data":"69d7fa839276f3e2043accb2fedf96947c18e1c9238f9132e14aeb7104424fb7"} Oct 30 13:12:51.350861 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:51.350735 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8746-b1fb9f" podUID="7f64527f-ec69-45a9-b9a3-99255f6e007e" containerName="test-curl-8746-b1fb9f" containerID="cri-o://316aab0dce90e66dcbd46d02434f423069eac2e551eae2fa24b416839697a1d7" gracePeriod=2 Oct 30 13:12:51.364646 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:51.364615 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-b1fb9f\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="7f64527f-ec69-45a9-b9a3-99255f6e007e" pod="opendatahub/test-curl-8746-b1fb9f" Oct 30 13:12:51.404157 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:51.404104 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8746-063bf8" podStartSLOduration=1.404085216 podStartE2EDuration="1.404085216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:51.40367516 +0000 UTC m=+1228.035862322" watchObservedRunningTime="2026-10-30 13:12:51.404085216 +0000 UTC m=+1228.036272379" Oct 30 13:12:51.425163 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:51.425111 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-b1fb9f\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="7f64527f-ec69-45a9-b9a3-99255f6e007e" pod="opendatahub/test-curl-8746-b1fb9f" Oct 30 13:12:51.538945 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:51.538919 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8746-263ede" Oct 30 13:12:51.658757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:51.658724 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9027bd5-2cde-4261-bd56-6054887c345f-kube-api-access-r27fx\" (UniqueName: \"kubernetes.io/projected/b9027bd5-2cde-4261-bd56-6054887c345f-kube-api-access-r27fx\") pod \"b9027bd5-2cde-4261-bd56-6054887c345f\" (UID: \"b9027bd5-2cde-4261-bd56-6054887c345f\") " Oct 30 13:12:51.660826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:51.660803 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9027bd5-2cde-4261-bd56-6054887c345f-kube-api-access-r27fx" pod "b9027bd5-2cde-4261-bd56-6054887c345f" (UID: "b9027bd5-2cde-4261-bd56-6054887c345f"). InnerVolumeSpecName "kube-api-access-r27fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:51.761038 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:51.761011 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r27fx\" (UniqueName: \"kubernetes.io/projected/b9027bd5-2cde-4261-bd56-6054887c345f-kube-api-access-r27fx\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:12:52.235143 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:52.235107 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8746-063bf8"] Oct 30 13:12:52.239460 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:52.239430 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8746-063bf8"] Oct 30 13:12:52.359725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:52.359686 7010 generic.go:356] "Generic (PLEG): container finished" podID="b9027bd5-2cde-4261-bd56-6054887c345f" containerID="ab649230cc57d5c9eaac2ee86f2ec00fd7ebf4b491b68070b5edd887da31d3cb" exitCode=137 Oct 30 13:12:52.359725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:52.359722 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8746-263ede" Oct 30 13:12:52.360045 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:52.359770 7010 scope.go:122] "RemoveContainer" containerID="ab649230cc57d5c9eaac2ee86f2ec00fd7ebf4b491b68070b5edd887da31d3cb" Oct 30 13:12:52.369778 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:52.369763 7010 scope.go:122] "RemoveContainer" containerID="ab649230cc57d5c9eaac2ee86f2ec00fd7ebf4b491b68070b5edd887da31d3cb" Oct 30 13:12:52.370043 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:52.370025 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab649230cc57d5c9eaac2ee86f2ec00fd7ebf4b491b68070b5edd887da31d3cb\": container with ID starting with ab649230cc57d5c9eaac2ee86f2ec00fd7ebf4b491b68070b5edd887da31d3cb not found: ID does not exist" containerID="ab649230cc57d5c9eaac2ee86f2ec00fd7ebf4b491b68070b5edd887da31d3cb" Oct 30 13:12:52.370092 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:52.370051 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab649230cc57d5c9eaac2ee86f2ec00fd7ebf4b491b68070b5edd887da31d3cb"} err="failed to get container status \"ab649230cc57d5c9eaac2ee86f2ec00fd7ebf4b491b68070b5edd887da31d3cb\": rpc error: code = NotFound desc = could not find container \"ab649230cc57d5c9eaac2ee86f2ec00fd7ebf4b491b68070b5edd887da31d3cb\": container with ID starting with ab649230cc57d5c9eaac2ee86f2ec00fd7ebf4b491b68070b5edd887da31d3cb not found: ID does not exist" Oct 30 13:12:52.382531 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:52.382476 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-263ede\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="b9027bd5-2cde-4261-bd56-6054887c345f" pod="opendatahub/test-curl-8746-263ede" Oct 30 13:12:52.385155 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:52.385132 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8746-9610d2"] Oct 30 13:12:52.385821 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:52.385806 7010 cpu_manager.go:414] "containerMap: removing container" podUID="deb9418e-f302-40db-bb61-264dab5d8875" containerName="test-curl-8746-063bf8" Oct 30 13:12:52.385877 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:52.385823 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb9418e-f302-40db-bb61-264dab5d8875" containerName="test-curl-8746-063bf8" Oct 30 13:12:52.385970 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:52.385960 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb9418e-f302-40db-bb61-264dab5d8875" containerName="test-curl-8746-063bf8" Oct 30 13:12:52.390940 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:52.390924 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8746-9610d2" Oct 30 13:12:52.409971 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:52.404862 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-263ede\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="b9027bd5-2cde-4261-bd56-6054887c345f" pod="opendatahub/test-curl-8746-263ede" Oct 30 13:12:52.409971 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:52.406575 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8746-9610d2"] Oct 30 13:12:52.572219 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:52.572137 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgk7k\" (UniqueName: \"kubernetes.io/projected/2df9f039-6193-46d6-aa67-117f7f221444-kube-api-access-mgk7k\") pod \"test-curl-8746-9610d2\" (UID: \"2df9f039-6193-46d6-aa67-117f7f221444\") " pod="opendatahub/test-curl-8746-9610d2" Oct 30 13:12:52.673606 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:52.673573 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgk7k\" (UniqueName: \"kubernetes.io/projected/2df9f039-6193-46d6-aa67-117f7f221444-kube-api-access-mgk7k\") pod \"test-curl-8746-9610d2\" (UID: \"2df9f039-6193-46d6-aa67-117f7f221444\") " pod="opendatahub/test-curl-8746-9610d2" Oct 30 13:12:52.703628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:52.703597 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgk7k\" (UniqueName: \"kubernetes.io/projected/2df9f039-6193-46d6-aa67-117f7f221444-kube-api-access-mgk7k\") pod \"test-curl-8746-9610d2\" (UID: \"2df9f039-6193-46d6-aa67-117f7f221444\") " pod="opendatahub/test-curl-8746-9610d2" Oct 30 13:12:52.731140 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:52.731113 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8746-9610d2" Oct 30 13:12:53.035816 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:53.035793 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8746-9610d2"] Oct 30 13:12:53.037203 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:12:53.037171 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2df9f039_6193_46d6_aa67_117f7f221444.slice/crio-6f7ca5c46b1fe49c9d138bfae126a4e533460a57da81f6489283e6e7c32262d5 WatchSource:0}: Error finding container 6f7ca5c46b1fe49c9d138bfae126a4e533460a57da81f6489283e6e7c32262d5: Status 404 returned error can't find the container with id 6f7ca5c46b1fe49c9d138bfae126a4e533460a57da81f6489283e6e7c32262d5 Oct 30 13:12:53.371736 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:53.371643 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8746-9610d2" event={"ID":"2df9f039-6193-46d6-aa67-117f7f221444","Type":"ContainerStarted","Data":"af6dca3067effae9d6e6b0cd63cabc221e1a0cba23cabd374b97fde65abd75fd"} Oct 30 13:12:53.371736 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:53.371680 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8746-9610d2" event={"ID":"2df9f039-6193-46d6-aa67-117f7f221444","Type":"ContainerStarted","Data":"6f7ca5c46b1fe49c9d138bfae126a4e533460a57da81f6489283e6e7c32262d5"} Oct 30 13:12:53.371851 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:53.371759 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8746-063bf8" podUID="deb9418e-f302-40db-bb61-264dab5d8875" containerName="test-curl-8746-063bf8" containerID="cri-o://6b920ddfc22cd40b9cd5e7c5e8d7575411a1e41e1f2e8e8c3e6b46402ba91f4a" gracePeriod=2 Oct 30 13:12:53.385961 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:53.385918 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-063bf8\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="deb9418e-f302-40db-bb61-264dab5d8875" pod="opendatahub/test-curl-8746-063bf8" Oct 30 13:12:53.399802 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:53.399759 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-263ede\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="b9027bd5-2cde-4261-bd56-6054887c345f" pod="opendatahub/test-curl-8746-263ede" Oct 30 13:12:53.440553 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:53.440467 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8746-9610d2" podStartSLOduration=1.440452544 podStartE2EDuration="1.440452544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:53.43828288 +0000 UTC m=+1230.070470040" watchObservedRunningTime="2026-10-30 13:12:53.440452544 +0000 UTC m=+1230.072639705" Oct 30 13:12:53.452505 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:53.452446 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-063bf8\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="deb9418e-f302-40db-bb61-264dab5d8875" pod="opendatahub/test-curl-8746-063bf8" Oct 30 13:12:53.466383 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:53.466339 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-263ede\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="b9027bd5-2cde-4261-bd56-6054887c345f" pod="opendatahub/test-curl-8746-263ede" Oct 30 13:12:53.492199 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:53.491378 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9027bd5-2cde-4261-bd56-6054887c345f" path="/var/lib/kubelet/pods/b9027bd5-2cde-4261-bd56-6054887c345f/volumes" Oct 30 13:12:53.501287 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:53.501246 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-063bf8\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="deb9418e-f302-40db-bb61-264dab5d8875" pod="opendatahub/test-curl-8746-063bf8" Oct 30 13:12:53.515148 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:53.515116 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-263ede\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="b9027bd5-2cde-4261-bd56-6054887c345f" pod="opendatahub/test-curl-8746-263ede" Oct 30 13:12:53.562003 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:53.561980 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8746-b1fb9f" Oct 30 13:12:53.576121 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:53.576084 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-b1fb9f\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="7f64527f-ec69-45a9-b9a3-99255f6e007e" pod="opendatahub/test-curl-8746-b1fb9f" Oct 30 13:12:53.687949 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:53.687920 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f64527f-ec69-45a9-b9a3-99255f6e007e-kube-api-access-djkmd\" (UniqueName: \"kubernetes.io/projected/7f64527f-ec69-45a9-b9a3-99255f6e007e-kube-api-access-djkmd\") pod \"7f64527f-ec69-45a9-b9a3-99255f6e007e\" (UID: \"7f64527f-ec69-45a9-b9a3-99255f6e007e\") " Oct 30 13:12:53.690189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:53.690165 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f64527f-ec69-45a9-b9a3-99255f6e007e-kube-api-access-djkmd" pod "7f64527f-ec69-45a9-b9a3-99255f6e007e" (UID: "7f64527f-ec69-45a9-b9a3-99255f6e007e"). InnerVolumeSpecName "kube-api-access-djkmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:53.789789 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:53.789760 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djkmd\" (UniqueName: \"kubernetes.io/projected/7f64527f-ec69-45a9-b9a3-99255f6e007e-kube-api-access-djkmd\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:12:54.342356 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:54.342268 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8746-9610d2"] Oct 30 13:12:54.350515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:54.350472 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8746-9610d2"] Oct 30 13:12:54.381386 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:54.381349 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8746-b1fb9f" Oct 30 13:12:54.381730 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:54.381694 7010 generic.go:356] "Generic (PLEG): container finished" podID="7f64527f-ec69-45a9-b9a3-99255f6e007e" containerID="316aab0dce90e66dcbd46d02434f423069eac2e551eae2fa24b416839697a1d7" exitCode=137 Oct 30 13:12:54.382058 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:54.382038 7010 scope.go:122] "RemoveContainer" containerID="316aab0dce90e66dcbd46d02434f423069eac2e551eae2fa24b416839697a1d7" Oct 30 13:12:54.400827 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:54.400550 7010 scope.go:122] "RemoveContainer" containerID="316aab0dce90e66dcbd46d02434f423069eac2e551eae2fa24b416839697a1d7" Oct 30 13:12:54.400827 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:54.400729 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-b1fb9f\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="7f64527f-ec69-45a9-b9a3-99255f6e007e" pod="opendatahub/test-curl-8746-b1fb9f" Oct 30 13:12:54.401296 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:54.401263 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316aab0dce90e66dcbd46d02434f423069eac2e551eae2fa24b416839697a1d7\": container with ID starting with 316aab0dce90e66dcbd46d02434f423069eac2e551eae2fa24b416839697a1d7 not found: ID does not exist" containerID="316aab0dce90e66dcbd46d02434f423069eac2e551eae2fa24b416839697a1d7" Oct 30 13:12:54.401391 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:54.401304 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316aab0dce90e66dcbd46d02434f423069eac2e551eae2fa24b416839697a1d7"} err="failed to get container status \"316aab0dce90e66dcbd46d02434f423069eac2e551eae2fa24b416839697a1d7\": rpc error: code = NotFound desc = could not find container \"316aab0dce90e66dcbd46d02434f423069eac2e551eae2fa24b416839697a1d7\": container with ID starting with 316aab0dce90e66dcbd46d02434f423069eac2e551eae2fa24b416839697a1d7 not found: ID does not exist" Oct 30 13:12:54.414841 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:54.414805 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-b1fb9f\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="7f64527f-ec69-45a9-b9a3-99255f6e007e" pod="opendatahub/test-curl-8746-b1fb9f" Oct 30 13:12:55.303539 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.303472 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57"] Oct 30 13:12:55.304477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.304456 7010 cpu_manager.go:414] "containerMap: removing container" podUID="2df9f039-6193-46d6-aa67-117f7f221444" containerName="test-curl-8746-9610d2" Oct 30 13:12:55.304477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.304477 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2df9f039-6193-46d6-aa67-117f7f221444" containerName="test-curl-8746-9610d2" Oct 30 13:12:55.304684 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.304659 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="2df9f039-6193-46d6-aa67-117f7f221444" containerName="test-curl-8746-9610d2" Oct 30 13:12:55.309757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.309738 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.328331 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:55.324724 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-b1fb9f\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="7f64527f-ec69-45a9-b9a3-99255f6e007e" pod="opendatahub/test-curl-8746-b1fb9f" Oct 30 13:12:55.328331 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.326599 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fafb3f-openshift-default-dockercfg-7mrg4\"" Oct 30 13:12:55.331770 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.331737 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57"] Oct 30 13:12:55.390189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.390155 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8746-9610d2" podUID="2df9f039-6193-46d6-aa67-117f7f221444" containerName="test-curl-8746-9610d2" containerID="cri-o://af6dca3067effae9d6e6b0cd63cabc221e1a0cba23cabd374b97fde65abd75fd" gracePeriod=2 Oct 30 13:12:55.418720 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:55.405735 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-b1fb9f\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="7f64527f-ec69-45a9-b9a3-99255f6e007e" pod="opendatahub/test-curl-8746-b1fb9f" Oct 30 13:12:55.471863 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:55.452720 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-9610d2\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="2df9f039-6193-46d6-aa67-117f7f221444" pod="opendatahub/test-curl-8746-9610d2" Oct 30 13:12:55.471863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.453765 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-data\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.471863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.453845 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-credential-socket\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.471863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.454037 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-podinfo\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.471863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.454083 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-token\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.471863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.454199 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-envoy\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.471863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.454225 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istiod-ca-cert\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.471863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.454256 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l75t9\" (UniqueName: \"kubernetes.io/projected/94ab8970-c142-48e4-9e5e-afbe6aa9f812-kube-api-access-l75t9\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.471863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.454307 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-workload-certs\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.471863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.454372 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-workload-socket\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.555604 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.555561 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-token\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.576728 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.576694 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-envoy\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.576940 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.576923 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istiod-ca-cert\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.577078 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.577063 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l75t9\" (UniqueName: \"kubernetes.io/projected/94ab8970-c142-48e4-9e5e-afbe6aa9f812-kube-api-access-l75t9\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.577225 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.577210 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-workload-certs\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.577390 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.577376 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-workload-socket\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.577550 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.577534 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-data\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.577734 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.577716 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-credential-socket\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.578094 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.578074 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-podinfo\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.598521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.592256 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f64527f-ec69-45a9-b9a3-99255f6e007e" path="/var/lib/kubelet/pods/7f64527f-ec69-45a9-b9a3-99255f6e007e/volumes" Oct 30 13:12:55.598521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.593065 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-workload-certs\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.598521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.593912 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-data\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.598521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.594333 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-workload-socket\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.598521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.595429 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istiod-ca-cert\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.598521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.595689 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-credential-socket\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.620529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.616937 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-envoy\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.655527 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.653408 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-podinfo\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.678568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.667458 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l75t9\" (UniqueName: \"kubernetes.io/projected/94ab8970-c142-48e4-9e5e-afbe6aa9f812-kube-api-access-l75t9\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.688110 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.688045 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-token\") pod \"e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.903879 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.903785 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8746-063bf8" Oct 30 13:12:55.935198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.935167 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:55.985068 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.985031 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb9418e-f302-40db-bb61-264dab5d8875-kube-api-access-l8d7v\" (UniqueName: \"kubernetes.io/projected/deb9418e-f302-40db-bb61-264dab5d8875-kube-api-access-l8d7v\") pod \"deb9418e-f302-40db-bb61-264dab5d8875\" (UID: \"deb9418e-f302-40db-bb61-264dab5d8875\") " Oct 30 13:12:55.987974 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:55.987946 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb9418e-f302-40db-bb61-264dab5d8875-kube-api-access-l8d7v" pod "deb9418e-f302-40db-bb61-264dab5d8875" (UID: "deb9418e-f302-40db-bb61-264dab5d8875"). InnerVolumeSpecName "kube-api-access-l8d7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:56.088126 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:56.088080 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8d7v\" (UniqueName: \"kubernetes.io/projected/deb9418e-f302-40db-bb61-264dab5d8875-kube-api-access-l8d7v\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:12:56.292811 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:56.291439 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57"] Oct 30 13:12:56.343510 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:12:56.340398 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ab8970_c142_48e4_9e5e_afbe6aa9f812.slice/crio-e2ca04b4f1189568c9ce79fd58997521c5e160bd92b5729407c65b76d838f0bc WatchSource:0}: Error finding container e2ca04b4f1189568c9ce79fd58997521c5e160bd92b5729407c65b76d838f0bc: Status 404 returned error can't find the container with id e2ca04b4f1189568c9ce79fd58997521c5e160bd92b5729407c65b76d838f0bc Oct 30 13:12:56.346677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:56.346529 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:56.346802 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:56.346734 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:56.346802 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:56.346778 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:12:56.404514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:56.400817 7010 generic.go:356] "Generic (PLEG): container finished" podID="deb9418e-f302-40db-bb61-264dab5d8875" containerID="6b920ddfc22cd40b9cd5e7c5e8d7575411a1e41e1f2e8e8c3e6b46402ba91f4a" exitCode=137 Oct 30 13:12:56.404514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:56.400882 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8746-063bf8" Oct 30 13:12:56.404514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:56.400890 7010 scope.go:122] "RemoveContainer" containerID="6b920ddfc22cd40b9cd5e7c5e8d7575411a1e41e1f2e8e8c3e6b46402ba91f4a" Oct 30 13:12:56.405120 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:56.405034 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" event={"ID":"94ab8970-c142-48e4-9e5e-afbe6aa9f812","Type":"ContainerStarted","Data":"e2ca04b4f1189568c9ce79fd58997521c5e160bd92b5729407c65b76d838f0bc"} Oct 30 13:12:56.428000 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:56.427976 7010 scope.go:122] "RemoveContainer" containerID="6b920ddfc22cd40b9cd5e7c5e8d7575411a1e41e1f2e8e8c3e6b46402ba91f4a" Oct 30 13:12:56.428355 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:56.428330 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b920ddfc22cd40b9cd5e7c5e8d7575411a1e41e1f2e8e8c3e6b46402ba91f4a\": container with ID starting with 6b920ddfc22cd40b9cd5e7c5e8d7575411a1e41e1f2e8e8c3e6b46402ba91f4a not found: ID does not exist" containerID="6b920ddfc22cd40b9cd5e7c5e8d7575411a1e41e1f2e8e8c3e6b46402ba91f4a" Oct 30 13:12:56.428441 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:56.428363 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b920ddfc22cd40b9cd5e7c5e8d7575411a1e41e1f2e8e8c3e6b46402ba91f4a"} err="failed to get container status \"6b920ddfc22cd40b9cd5e7c5e8d7575411a1e41e1f2e8e8c3e6b46402ba91f4a\": rpc error: code = NotFound desc = could not find container \"6b920ddfc22cd40b9cd5e7c5e8d7575411a1e41e1f2e8e8c3e6b46402ba91f4a\": container with ID starting with 6b920ddfc22cd40b9cd5e7c5e8d7575411a1e41e1f2e8e8c3e6b46402ba91f4a not found: ID does not exist" Oct 30 13:12:56.456894 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:56.456856 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-063bf8\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="deb9418e-f302-40db-bb61-264dab5d8875" pod="opendatahub/test-curl-8746-063bf8" Oct 30 13:12:57.414018 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:57.413991 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" event={"ID":"94ab8970-c142-48e4-9e5e-afbe6aa9f812","Type":"ContainerStarted","Data":"12380aab9b447818abc37eb063a71268038643f79d2972e3eb5b12e41841ccb7"} Oct 30 13:12:57.428362 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:57.428334 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-063bf8\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="deb9418e-f302-40db-bb61-264dab5d8875" pod="opendatahub/test-curl-8746-063bf8" Oct 30 13:12:57.489728 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:57.489698 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb9418e-f302-40db-bb61-264dab5d8875" path="/var/lib/kubelet/pods/deb9418e-f302-40db-bb61-264dab5d8875/volumes" Oct 30 13:12:57.515845 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:57.515785 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" podStartSLOduration=2.515766255 podStartE2EDuration="2.515766255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:57.500217396 +0000 UTC m=+1234.132404610" watchObservedRunningTime="2026-10-30 13:12:57.515766255 +0000 UTC m=+1234.147953418" Oct 30 13:12:57.590688 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:57.590663 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8746-9610d2" Oct 30 13:12:57.742192 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:57.742097 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df9f039-6193-46d6-aa67-117f7f221444-kube-api-access-mgk7k\" (UniqueName: \"kubernetes.io/projected/2df9f039-6193-46d6-aa67-117f7f221444-kube-api-access-mgk7k\") pod \"2df9f039-6193-46d6-aa67-117f7f221444\" (UID: \"2df9f039-6193-46d6-aa67-117f7f221444\") " Oct 30 13:12:57.744246 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:57.744214 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df9f039-6193-46d6-aa67-117f7f221444-kube-api-access-mgk7k" pod "2df9f039-6193-46d6-aa67-117f7f221444" (UID: "2df9f039-6193-46d6-aa67-117f7f221444"). InnerVolumeSpecName "kube-api-access-mgk7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:12:57.844933 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:57.844896 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgk7k\" (UniqueName: \"kubernetes.io/projected/2df9f039-6193-46d6-aa67-117f7f221444-kube-api-access-mgk7k\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:12:57.935547 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:57.935512 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:57.941256 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:57.941233 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:58.452705 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:58.450420 7010 generic.go:356] "Generic (PLEG): container finished" podID="2df9f039-6193-46d6-aa67-117f7f221444" containerID="af6dca3067effae9d6e6b0cd63cabc221e1a0cba23cabd374b97fde65abd75fd" exitCode=137 Oct 30 13:12:58.452705 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:58.451984 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8746-9610d2" Oct 30 13:12:58.453229 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:58.452737 7010 scope.go:122] "RemoveContainer" containerID="af6dca3067effae9d6e6b0cd63cabc221e1a0cba23cabd374b97fde65abd75fd" Oct 30 13:12:58.453229 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:58.452746 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:58.459576 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:58.456954 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:12:58.466353 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:58.466321 7010 scope.go:122] "RemoveContainer" containerID="af6dca3067effae9d6e6b0cd63cabc221e1a0cba23cabd374b97fde65abd75fd" Oct 30 13:12:58.466857 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:58.466775 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af6dca3067effae9d6e6b0cd63cabc221e1a0cba23cabd374b97fde65abd75fd\": container with ID starting with af6dca3067effae9d6e6b0cd63cabc221e1a0cba23cabd374b97fde65abd75fd not found: ID does not exist" containerID="af6dca3067effae9d6e6b0cd63cabc221e1a0cba23cabd374b97fde65abd75fd" Oct 30 13:12:58.466857 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:58.466810 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af6dca3067effae9d6e6b0cd63cabc221e1a0cba23cabd374b97fde65abd75fd"} err="failed to get container status \"af6dca3067effae9d6e6b0cd63cabc221e1a0cba23cabd374b97fde65abd75fd\": rpc error: code = NotFound desc = could not find container \"af6dca3067effae9d6e6b0cd63cabc221e1a0cba23cabd374b97fde65abd75fd\": container with ID starting with af6dca3067effae9d6e6b0cd63cabc221e1a0cba23cabd374b97fde65abd75fd not found: ID does not exist" Oct 30 13:12:58.533536 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:58.526360 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8746-9610d2\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="2df9f039-6193-46d6-aa67-117f7f221444" pod="opendatahub/test-curl-8746-9610d2" Oct 30 13:12:59.401033 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:12:59.400989 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:12:59.488941 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:12:59.488909 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2df9f039-6193-46d6-aa67-117f7f221444" path="/var/lib/kubelet/pods/2df9f039-6193-46d6-aa67-117f7f221444/volumes" Oct 30 13:13:07.778436 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.778401 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf"] Oct 30 13:13:07.784193 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.784168 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.811329 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.811300 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-df6c3661\"/\"e2e-lifecycle-model-df6c3661-kserve-self-signed-certs\"" Oct 30 13:13:07.811776 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.811735 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-df6c3661\"/\"default-dockercfg-rfh6f\"" Oct 30 13:13:07.821896 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.821687 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-df6c3661\"/\"kube-root-ca.crt\"" Oct 30 13:13:07.821896 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.821774 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-df6c3661\"/\"openshift-service-ca.crt\"" Oct 30 13:13:07.822552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.822527 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf"] Oct 30 13:13:07.824927 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.824904 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8042b023-65f6-4cc1-a2b6-5f94eead7207-tls-certs\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.825092 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.825071 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-model-cache\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.825232 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.825213 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-dshm\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.825301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.825282 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfbzl\" (UniqueName: \"kubernetes.io/projected/8042b023-65f6-4cc1-a2b6-5f94eead7207-kube-api-access-lfbzl\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.825362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.825348 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-home\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.825583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.825563 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-tmp-dir\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.927805 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.927759 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-model-cache\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.928018 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.927896 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-dshm\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.928018 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.927945 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbzl\" (UniqueName: \"kubernetes.io/projected/8042b023-65f6-4cc1-a2b6-5f94eead7207-kube-api-access-lfbzl\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.928018 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.928005 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-home\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.928181 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.928170 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-tmp-dir\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.928427 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.928227 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-model-cache\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.928427 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.928242 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8042b023-65f6-4cc1-a2b6-5f94eead7207-tls-certs\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.928656 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.928454 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-tmp-dir\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.928758 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.928725 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-home\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.932096 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.932054 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-dshm\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.932335 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.932315 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8042b023-65f6-4cc1-a2b6-5f94eead7207-tls-certs\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:07.951994 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:07.951966 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfbzl\" (UniqueName: \"kubernetes.io/projected/8042b023-65f6-4cc1-a2b6-5f94eead7207-kube-api-access-lfbzl\") pod \"e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:08.134845 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:08.134800 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:09.410829 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:09.410797 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf"] Oct 30 13:13:09.655957 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:09.655910 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" event={"ID":"8042b023-65f6-4cc1-a2b6-5f94eead7207","Type":"ContainerStarted","Data":"78796a6a59cfc71492b4913e04184f36fe56a9d74f933c61e9d6141183d2bc55"} Oct 30 13:13:10.731998 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:10.731958 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" event={"ID":"8042b023-65f6-4cc1-a2b6-5f94eead7207","Type":"ContainerStarted","Data":"36f5ff33515a43ed760a12e3872f91c61b34c9f9a24e34a9e67b148f3af94a0a"} Oct 30 13:13:10.856512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:10.838940 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" podStartSLOduration=3.8389210719999998 podStartE2EDuration="3.838921072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:10.83726157 +0000 UTC m=+1247.469448733" watchObservedRunningTime="2026-10-30 13:13:10.838921072 +0000 UTC m=+1247.471108239" Oct 30 13:13:11.141606 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:11.141284 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:11.941520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:11.927613 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:12.035005 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:12.034937 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:12.037154 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:12.037106 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:12.038033 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:12.037930 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 13:13:12.252827 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:12.252730 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 13:13:18.104661 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:18.104620 7010 patch_prober.go:29] interesting pod/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:13:18.105901 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:18.104696 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" podUID="428e36fe-a829-4bee-bcd5-5150d5358411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:13:18.135770 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:18.135742 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:18.148663 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:18.148631 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:18.964772 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:18.964740 7010 generic.go:356] "Generic (PLEG): container finished" podID="428e36fe-a829-4bee-bcd5-5150d5358411" containerID="7360cc0b840e330e07e05af9d2af3e4e62d4efc51841d2b2034a81ca71be709f" exitCode=0 Oct 30 13:13:18.964984 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:18.964819 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" event={"ID":"428e36fe-a829-4bee-bcd5-5150d5358411","Type":"ContainerDied","Data":"7360cc0b840e330e07e05af9d2af3e4e62d4efc51841d2b2034a81ca71be709f"} Oct 30 13:13:18.965372 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:18.965353 7010 scope.go:122] "RemoveContainer" containerID="7360cc0b840e330e07e05af9d2af3e4e62d4efc51841d2b2034a81ca71be709f" Oct 30 13:13:18.967290 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:18.967258 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:13:18.967382 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:18.967344 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:13:18.967423 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:18.967405 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:13:18.977590 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:18.977569 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:13:19.974073 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:19.974037 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" event={"ID":"428e36fe-a829-4bee-bcd5-5150d5358411","Type":"ContainerStarted","Data":"e8ec7090dd84e413dde6d4f90fda910a406f0c2039b9191bc45251a063290999"} Oct 30 13:13:20.786516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:20.784684 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:13:20.786516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:20.785759 7010 patch_prober.go:29] interesting pod/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:13:20.786516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:20.785810 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" podUID="428e36fe-a829-4bee-bcd5-5150d5358411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 13:13:21.782042 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:21.782002 7010 patch_prober.go:29] interesting pod/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:13:21.782444 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:21.782062 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" podUID="428e36fe-a829-4bee-bcd5-5150d5358411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 13:13:22.782075 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:22.782039 7010 patch_prober.go:29] interesting pod/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:13:22.782465 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:22.782097 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" podUID="428e36fe-a829-4bee-bcd5-5150d5358411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 13:13:23.782416 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:23.782376 7010 patch_prober.go:29] interesting pod/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:13:23.782870 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:23.782470 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" podUID="428e36fe-a829-4bee-bcd5-5150d5358411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 13:13:24.781832 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:24.781796 7010 patch_prober.go:29] interesting pod/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 13:13:24.782022 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:24.781848 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" podUID="428e36fe-a829-4bee-bcd5-5150d5358411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 13:13:25.786120 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:25.786082 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:13:25.786640 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:25.786478 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:13:25.787271 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:25.787249 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:13:29.410815 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:13:29.410774 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:13:40.708693 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:40.708598 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45"] Oct 30 13:13:40.712332 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:40.712295 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:40.748366 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:40.748332 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-659d72-openshift-default-dockercfg-pnc2s\"" Oct 30 13:13:40.762259 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:40.762221 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45"] Oct 30 13:13:40.843471 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:40.843422 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-token\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:40.843774 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:40.843751 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-data\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:40.844226 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:40.844198 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-credential-socket\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:40.844683 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:40.844616 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6v6h\" (UniqueName: \"kubernetes.io/projected/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-kube-api-access-w6v6h\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:40.853508 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:40.844843 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-workload-socket\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:40.873520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:40.862430 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istiod-ca-cert\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:40.873848 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:40.873820 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-envoy\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:40.874030 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:40.874014 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-workload-certs\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:40.874134 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:40.874120 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-podinfo\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:40.986199 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:40.986076 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istiod-ca-cert\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.004812 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.004663 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istiod-ca-cert\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.005540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.005512 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-envoy\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.005815 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.005796 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-workload-certs\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.006400 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.006376 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-podinfo\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.006752 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.006726 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-token\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.007444 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.006326 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-workload-certs\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.007982 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.007963 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-data\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.008131 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.007409 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-data\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.009471 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.009441 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-credential-socket\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.009689 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.008437 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-credential-socket\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.010258 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.010237 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6v6h\" (UniqueName: \"kubernetes.io/projected/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-kube-api-access-w6v6h\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.021963 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.021685 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-workload-socket\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.022678 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.022564 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-workload-socket\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.026264 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.026235 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-podinfo\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.044941 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.044864 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-token\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.057442 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.057318 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6v6h\" (UniqueName: \"kubernetes.io/projected/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-kube-api-access-w6v6h\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.079513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.076819 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-envoy\") pod \"e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.338169 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.338074 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:41.680017 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.679983 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45"] Oct 30 13:13:41.681818 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:13:41.681772 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0628c83b_2550_4bd0_aa9f_e0a8c869f0bb.slice/crio-8b60770071465e04a424a28165b1691f6457fd72284fc28041e0bf42b78f5449 WatchSource:0}: Error finding container 8b60770071465e04a424a28165b1691f6457fd72284fc28041e0bf42b78f5449: Status 404 returned error can't find the container with id 8b60770071465e04a424a28165b1691f6457fd72284fc28041e0bf42b78f5449 Oct 30 13:13:41.684840 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.684801 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:13:41.684944 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.684874 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:13:41.684944 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:41.684905 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:13:42.233752 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:42.233715 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" event={"ID":"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb","Type":"ContainerStarted","Data":"2b2e23a440240fe691fedecbf6d296595183aa9701365b04891203b93b90f843"} Oct 30 13:13:42.234310 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:42.234276 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" event={"ID":"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb","Type":"ContainerStarted","Data":"8b60770071465e04a424a28165b1691f6457fd72284fc28041e0bf42b78f5449"} Oct 30 13:13:42.293532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:42.293443 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" podStartSLOduration=2.29342873 podStartE2EDuration="2.29342873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:42.291780613 +0000 UTC m=+1278.923967774" watchObservedRunningTime="2026-10-30 13:13:42.29342873 +0000 UTC m=+1278.925615892" Oct 30 13:13:42.338573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:42.338536 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:42.345754 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:42.345722 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:43.272072 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:43.272043 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:43.275131 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:43.275108 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:13:52.310795 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:52.310762 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz"] Oct 30 13:13:52.314433 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:52.314414 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:13:52.340179 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:52.340150 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-b29df4-serving-cert\"" Oct 30 13:13:52.358954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:52.358914 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz"] Oct 30 13:13:52.394167 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:52.394127 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa8a642-6cd9-4f8c-889f-462088c95ecf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz\" (UID: \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:13:52.394429 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:52.394407 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8skx6\" (UniqueName: \"kubernetes.io/projected/8fa8a642-6cd9-4f8c-889f-462088c95ecf-kube-api-access-8skx6\") pod \"maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz\" (UID: \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:13:52.394555 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:52.394455 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa8a642-6cd9-4f8c-889f-462088c95ecf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz\" (UID: \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:13:52.394673 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:52.394652 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fa8a642-6cd9-4f8c-889f-462088c95ecf-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz\" (UID: \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:13:52.496471 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:52.496438 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skx6\" (UniqueName: \"kubernetes.io/projected/8fa8a642-6cd9-4f8c-889f-462088c95ecf-kube-api-access-8skx6\") pod \"maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz\" (UID: \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:13:52.496669 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:52.496479 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa8a642-6cd9-4f8c-889f-462088c95ecf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz\" (UID: \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:13:52.496669 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:52.496594 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fa8a642-6cd9-4f8c-889f-462088c95ecf-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz\" (UID: \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:13:52.496779 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:52.496688 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa8a642-6cd9-4f8c-889f-462088c95ecf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz\" (UID: \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:13:52.496841 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:13:52.496808 7010 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-serving-cert: secret "maas-api-e2e-shared-b-w4-b29df4-serving-cert" not found Oct 30 13:13:52.496895 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:13:52.496873 7010 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fa8a642-6cd9-4f8c-889f-462088c95ecf-maas-api-tls podName:8fa8a642-6cd9-4f8c-889f-462088c95ecf nodeName:}" failed. No retries permitted until 2026-10-30 13:13:52.996856852 +0000 UTC m=+1289.629044008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8fa8a642-6cd9-4f8c-889f-462088c95ecf-maas-api-tls") pod "maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" (UID: "8fa8a642-6cd9-4f8c-889f-462088c95ecf") : secret "maas-api-e2e-shared-b-w4-b29df4-serving-cert" not found Oct 30 13:13:52.497235 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:52.497216 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa8a642-6cd9-4f8c-889f-462088c95ecf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz\" (UID: \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:13:52.499040 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:52.499018 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fa8a642-6cd9-4f8c-889f-462088c95ecf-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz\" (UID: \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:13:52.552878 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:52.552841 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8skx6\" (UniqueName: \"kubernetes.io/projected/8fa8a642-6cd9-4f8c-889f-462088c95ecf-kube-api-access-8skx6\") pod \"maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz\" (UID: \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:13:53.005428 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:53.005391 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa8a642-6cd9-4f8c-889f-462088c95ecf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz\" (UID: \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:13:53.007911 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:53.007887 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa8a642-6cd9-4f8c-889f-462088c95ecf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz\" (UID: \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:13:53.228739 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:53.228702 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:13:53.616840 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:53.616809 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz"] Oct 30 13:13:53.618008 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:13:53.617972 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa8a642_6cd9_4f8c_889f_462088c95ecf.slice/crio-3981adc34a0d6b99a0bdd6347f789aedddf0009f4797f75291f016b420e97c0b WatchSource:0}: Error finding container 3981adc34a0d6b99a0bdd6347f789aedddf0009f4797f75291f016b420e97c0b: Status 404 returned error can't find the container with id 3981adc34a0d6b99a0bdd6347f789aedddf0009f4797f75291f016b420e97c0b Oct 30 13:13:54.364540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:54.364441 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" event={"ID":"8fa8a642-6cd9-4f8c-889f-462088c95ecf","Type":"ContainerStarted","Data":"be4bd166b9247c938475a7d9cfe8dc7e3521321bd6301bccdb2aa52c00a5e63d"} Oct 30 13:13:54.364540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:54.364523 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" event={"ID":"8fa8a642-6cd9-4f8c-889f-462088c95ecf","Type":"ContainerStarted","Data":"3981adc34a0d6b99a0bdd6347f789aedddf0009f4797f75291f016b420e97c0b"} Oct 30 13:13:54.364752 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:54.364672 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:13:54.428121 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:13:54.427976 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" podStartSLOduration=2.00334128 podStartE2EDuration="2.427951463s" totalImagesPullingTime="424.610183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:54.417091735 +0000 UTC m=+1291.049278906" watchObservedRunningTime="2026-10-30 13:13:54.427951463 +0000 UTC m=+1291.060138642" Oct 30 13:13:59.403304 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:13:59.403267 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:14:00.415689 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:00.414827 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:14:10.561014 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:10.560913 7010 generic.go:356] "Generic (PLEG): container finished" podID="94ab8970-c142-48e4-9e5e-afbe6aa9f812" containerID="12380aab9b447818abc37eb063a71268038643f79d2972e3eb5b12e41841ccb7" exitCode=0 Oct 30 13:14:10.561014 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:10.560987 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" event={"ID":"94ab8970-c142-48e4-9e5e-afbe6aa9f812","Type":"ContainerDied","Data":"12380aab9b447818abc37eb063a71268038643f79d2972e3eb5b12e41841ccb7"} Oct 30 13:14:10.561517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:10.561474 7010 scope.go:122] "RemoveContainer" containerID="12380aab9b447818abc37eb063a71268038643f79d2972e3eb5b12e41841ccb7" Oct 30 13:14:10.563324 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:10.563300 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:10.563401 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:10.563366 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:10.563444 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:10.563398 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:11.569901 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:11.569858 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" event={"ID":"94ab8970-c142-48e4-9e5e-afbe6aa9f812","Type":"ContainerStarted","Data":"15fb9fa8a4716d55dcceade979af6f3912d220f3b0a794a876e63a6ca844cfbe"} Oct 30 13:14:11.935626 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:11.935586 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:14:11.936719 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:11.936693 7010 patch_prober.go:29] interesting pod/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:14:11.936790 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:11.936750 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" podUID="94ab8970-c142-48e4-9e5e-afbe6aa9f812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 13:14:12.936232 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:12.936197 7010 patch_prober.go:29] interesting pod/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:14:12.936664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:12.936256 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" podUID="94ab8970-c142-48e4-9e5e-afbe6aa9f812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 13:14:13.452595 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:13.452558 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:14:13.935739 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:13.935707 7010 patch_prober.go:29] interesting pod/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:14:13.935990 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:13.935758 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" podUID="94ab8970-c142-48e4-9e5e-afbe6aa9f812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 13:14:14.936671 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:14.936625 7010 patch_prober.go:29] interesting pod/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:14:14.937182 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:14.936701 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" podUID="94ab8970-c142-48e4-9e5e-afbe6aa9f812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 13:14:15.936064 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:15.936027 7010 patch_prober.go:29] interesting pod/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:14:15.936264 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:15.936087 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" podUID="94ab8970-c142-48e4-9e5e-afbe6aa9f812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 13:14:16.935535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:16.935482 7010 patch_prober.go:29] interesting pod/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 13:14:16.935934 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:16.935558 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" podUID="94ab8970-c142-48e4-9e5e-afbe6aa9f812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 13:14:17.940562 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:17.940529 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:14:17.941717 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:17.941695 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:14:23.141931 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.141894 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729"] Oct 30 13:14:23.156753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.156726 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.169768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.169738 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729"] Oct 30 13:14:23.179533 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.176447 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-9c6033-openshift-default-dockercfg-5nxq8\"" Oct 30 13:14:23.343510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.342340 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-envoy\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.343510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.342461 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-podinfo\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.343510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.342518 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-workload-socket\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.343510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.342553 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-token\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.343510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.342615 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-workload-certs\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.343510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.342666 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3575f061-f8d0-4915-b4e8-c88c882407e5-istiod-ca-cert\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.343510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.342693 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-credential-socket\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.343510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.342753 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-data\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.343510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.342964 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrxkp\" (UniqueName: \"kubernetes.io/projected/3575f061-f8d0-4915-b4e8-c88c882407e5-kube-api-access-zrxkp\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.454556 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.448593 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-podinfo\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.454556 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.448661 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-workload-socket\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.454556 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.448694 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-token\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.454556 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.448761 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-workload-certs\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.454556 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.448815 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3575f061-f8d0-4915-b4e8-c88c882407e5-istiod-ca-cert\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.454556 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.448842 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-credential-socket\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.454556 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.448900 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-data\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.454556 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.449106 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrxkp\" (UniqueName: \"kubernetes.io/projected/3575f061-f8d0-4915-b4e8-c88c882407e5-kube-api-access-zrxkp\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.454556 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.449217 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-envoy\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.454556 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.450135 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-workload-certs\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.455263 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.455200 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-envoy\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.455263 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.455232 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-data\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.462510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.455467 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-workload-socket\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.462510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.455942 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3575f061-f8d0-4915-b4e8-c88c882407e5-istiod-ca-cert\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.462510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.456388 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-credential-socket\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.482517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.482461 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-podinfo\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.490967 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.488533 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrxkp\" (UniqueName: \"kubernetes.io/projected/3575f061-f8d0-4915-b4e8-c88c882407e5-kube-api-access-zrxkp\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.491394 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.491156 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-token\") pod \"e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:23.789025 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.788939 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-9c6033-openshift-default-dockercfg-5nxq8\"" Oct 30 13:14:23.794240 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:23.794209 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:24.344943 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:24.344915 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729"] Oct 30 13:14:24.347377 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:14:24.347351 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3575f061_f8d0_4915_b4e8_c88c882407e5.slice/crio-86da82f99c287fe240536c3535e410db85c5359ff1a01897c5bc3dd996dc5eae WatchSource:0}: Error finding container 86da82f99c287fe240536c3535e410db85c5359ff1a01897c5bc3dd996dc5eae: Status 404 returned error can't find the container with id 86da82f99c287fe240536c3535e410db85c5359ff1a01897c5bc3dd996dc5eae Oct 30 13:14:24.350215 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:24.350180 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:24.350308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:24.350249 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:24.350308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:24.350278 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:24.686469 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:24.686370 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" event={"ID":"3575f061-f8d0-4915-b4e8-c88c882407e5","Type":"ContainerStarted","Data":"d4629e7d65510a170b67d2a8e9336306456a592326d8dd6387f9ee30a3a4a799"} Oct 30 13:14:24.686469 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:24.686424 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" event={"ID":"3575f061-f8d0-4915-b4e8-c88c882407e5","Type":"ContainerStarted","Data":"86da82f99c287fe240536c3535e410db85c5359ff1a01897c5bc3dd996dc5eae"} Oct 30 13:14:24.793874 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:24.793796 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" podStartSLOduration=1.793773833 podStartE2EDuration="1.793773833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:24.778076916 +0000 UTC m=+1321.410264105" watchObservedRunningTime="2026-10-30 13:14:24.793773833 +0000 UTC m=+1321.425960996" Oct 30 13:14:24.794371 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:24.794349 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:25.807842 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:25.807811 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:26.710952 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:26.710914 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:26.712044 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:26.712021 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:14:29.404818 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:14:29.404776 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:14:32.680829 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:32.680785 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8743-a0b3da"] Oct 30 13:14:32.685624 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:32.685586 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8743-a0b3da" Oct 30 13:14:32.704913 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:32.704884 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8743-a0b3da"] Oct 30 13:14:32.713914 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:32.713878 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nqkjx\"" Oct 30 13:14:32.844508 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:32.844461 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxq9q\" (UniqueName: \"kubernetes.io/projected/72896412-b187-41b0-a942-ab1bad9f8fe9-kube-api-access-hxq9q\") pod \"test-curl-8743-a0b3da\" (UID: \"72896412-b187-41b0-a942-ab1bad9f8fe9\") " pod="opendatahub/test-curl-8743-a0b3da" Oct 30 13:14:32.946363 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:32.946277 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxq9q\" (UniqueName: \"kubernetes.io/projected/72896412-b187-41b0-a942-ab1bad9f8fe9-kube-api-access-hxq9q\") pod \"test-curl-8743-a0b3da\" (UID: \"72896412-b187-41b0-a942-ab1bad9f8fe9\") " pod="opendatahub/test-curl-8743-a0b3da" Oct 30 13:14:32.970573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:32.970539 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxq9q\" (UniqueName: \"kubernetes.io/projected/72896412-b187-41b0-a942-ab1bad9f8fe9-kube-api-access-hxq9q\") pod \"test-curl-8743-a0b3da\" (UID: \"72896412-b187-41b0-a942-ab1bad9f8fe9\") " pod="opendatahub/test-curl-8743-a0b3da" Oct 30 13:14:33.006498 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.006466 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8743-a0b3da" Oct 30 13:14:33.320615 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.320506 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8743-a0b3da"] Oct 30 13:14:33.320986 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:14:33.320954 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72896412_b187_41b0_a942_ab1bad9f8fe9.slice/crio-638ce0675bf8f3e80e7c0abf075468afc4ca7e4fec0c34572bdf4084a3ebac4e WatchSource:0}: Error finding container 638ce0675bf8f3e80e7c0abf075468afc4ca7e4fec0c34572bdf4084a3ebac4e: Status 404 returned error can't find the container with id 638ce0675bf8f3e80e7c0abf075468afc4ca7e4fec0c34572bdf4084a3ebac4e Oct 30 13:14:33.362961 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.362918 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb"] Oct 30 13:14:33.373534 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.370482 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.397437 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.397406 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8af0fe-openshift-default-dockercfg-txtlv\"" Oct 30 13:14:33.403944 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.403915 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb"] Oct 30 13:14:33.456982 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.456946 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-podinfo\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.457176 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.457140 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-token\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.457278 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.457210 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-workload-certs\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.457278 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.457248 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-credential-socket\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.457398 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.457372 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-data\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.457617 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.457594 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdmxf\" (UniqueName: \"kubernetes.io/projected/c0a3ed5e-a26f-4514-921d-24b409b1dc27-kube-api-access-tdmxf\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.457679 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.457666 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istiod-ca-cert\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.457778 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.457753 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-envoy\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.457842 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.457817 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-workload-socket\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.570479 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.568795 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-podinfo\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.570479 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.568915 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-token\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.570479 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.568960 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-workload-certs\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.570479 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.568993 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-credential-socket\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.570479 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.569038 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-data\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.570479 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.569176 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdmxf\" (UniqueName: \"kubernetes.io/projected/c0a3ed5e-a26f-4514-921d-24b409b1dc27-kube-api-access-tdmxf\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.570479 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.569238 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istiod-ca-cert\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.570479 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.569269 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-envoy\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.570479 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.569315 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-workload-socket\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.570479 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.570030 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-workload-socket\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.571358 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.571329 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-data\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.571936 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.571913 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istiod-ca-cert\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.572088 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.571964 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-workload-certs\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.572213 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.572190 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-credential-socket\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.583982 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.583948 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-envoy\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.593520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.591018 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-podinfo\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.628481 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.623752 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-token\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.650541 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.640302 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdmxf\" (UniqueName: \"kubernetes.io/projected/c0a3ed5e-a26f-4514-921d-24b409b1dc27-kube-api-access-tdmxf\") pod \"e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.735844 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.735801 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:33.924509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.920543 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8743-a0b3da" event={"ID":"72896412-b187-41b0-a942-ab1bad9f8fe9","Type":"ContainerStarted","Data":"40a7c21d0deb29df945cfb3cdc8143a152b17a2446b2cb5e43830dd151bf43e1"} Oct 30 13:14:33.924509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:33.920611 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8743-a0b3da" event={"ID":"72896412-b187-41b0-a942-ab1bad9f8fe9","Type":"ContainerStarted","Data":"638ce0675bf8f3e80e7c0abf075468afc4ca7e4fec0c34572bdf4084a3ebac4e"} Oct 30 13:14:34.300637 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:34.300532 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8743-a0b3da" podStartSLOduration=2.300513183 podStartE2EDuration="2.300513183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:34.005971255 +0000 UTC m=+1330.638158434" watchObservedRunningTime="2026-10-30 13:14:34.300513183 +0000 UTC m=+1330.932700346" Oct 30 13:14:34.304600 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:14:34.304555 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a3ed5e_a26f_4514_921d_24b409b1dc27.slice/crio-7a430f7c2931c1be0811c4c0afef6503fcaccfa0fc0867bc5f92d955a499eb93 WatchSource:0}: Error finding container 7a430f7c2931c1be0811c4c0afef6503fcaccfa0fc0867bc5f92d955a499eb93: Status 404 returned error can't find the container with id 7a430f7c2931c1be0811c4c0afef6503fcaccfa0fc0867bc5f92d955a499eb93 Oct 30 13:14:34.305868 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:34.305839 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb"] Oct 30 13:14:34.308129 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:34.308093 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:34.308205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:34.308179 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:34.308249 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:34.308224 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:14:34.912807 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:34.912767 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8743-a0b3da"] Oct 30 13:14:34.917955 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:34.917911 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8743-a0b3da"] Oct 30 13:14:34.937531 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:34.934409 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" event={"ID":"c0a3ed5e-a26f-4514-921d-24b409b1dc27","Type":"ContainerStarted","Data":"bb1e7b1a7e514b51b265d1829973e1c40f07091431f97f892bd5d5df44849fe0"} Oct 30 13:14:34.937531 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:34.934482 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" event={"ID":"c0a3ed5e-a26f-4514-921d-24b409b1dc27","Type":"ContainerStarted","Data":"7a430f7c2931c1be0811c4c0afef6503fcaccfa0fc0867bc5f92d955a499eb93"} Oct 30 13:14:35.737000 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:35.736964 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:35.744190 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:35.744164 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:35.870516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:35.868961 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" podStartSLOduration=2.868940113 podStartE2EDuration="2.868940113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:35.009640065 +0000 UTC m=+1331.641827244" watchObservedRunningTime="2026-10-30 13:14:35.868940113 +0000 UTC m=+1332.501127274" Oct 30 13:14:35.941110 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:35.941076 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8743-a0b3da" podUID="72896412-b187-41b0-a942-ab1bad9f8fe9" containerName="test-curl-8743-a0b3da" containerID="cri-o://40a7c21d0deb29df945cfb3cdc8143a152b17a2446b2cb5e43830dd151bf43e1" gracePeriod=2 Oct 30 13:14:35.941599 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:35.941176 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:35.942171 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:35.942150 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:14:37.958696 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:37.958662 7010 generic.go:356] "Generic (PLEG): container finished" podID="72896412-b187-41b0-a942-ab1bad9f8fe9" containerID="40a7c21d0deb29df945cfb3cdc8143a152b17a2446b2cb5e43830dd151bf43e1" exitCode=137 Oct 30 13:14:38.132724 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:38.132691 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8743-a0b3da" Oct 30 13:14:38.217058 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:38.216967 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72896412-b187-41b0-a942-ab1bad9f8fe9-kube-api-access-hxq9q\" (UniqueName: \"kubernetes.io/projected/72896412-b187-41b0-a942-ab1bad9f8fe9-kube-api-access-hxq9q\") pod \"72896412-b187-41b0-a942-ab1bad9f8fe9\" (UID: \"72896412-b187-41b0-a942-ab1bad9f8fe9\") " Oct 30 13:14:38.219113 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:38.219087 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72896412-b187-41b0-a942-ab1bad9f8fe9-kube-api-access-hxq9q" pod "72896412-b187-41b0-a942-ab1bad9f8fe9" (UID: "72896412-b187-41b0-a942-ab1bad9f8fe9"). InnerVolumeSpecName "kube-api-access-hxq9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:14:38.320072 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:38.320040 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxq9q\" (UniqueName: \"kubernetes.io/projected/72896412-b187-41b0-a942-ab1bad9f8fe9-kube-api-access-hxq9q\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:14:38.967972 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:38.967939 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8743-a0b3da" Oct 30 13:14:38.968544 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:38.967990 7010 scope.go:122] "RemoveContainer" containerID="40a7c21d0deb29df945cfb3cdc8143a152b17a2446b2cb5e43830dd151bf43e1" Oct 30 13:14:38.990960 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:14:38.990924 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8743-a0b3da\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="72896412-b187-41b0-a942-ab1bad9f8fe9" pod="opendatahub/test-curl-8743-a0b3da" Oct 30 13:14:39.414218 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:39.414183 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk"] Oct 30 13:14:39.414910 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:39.414895 7010 cpu_manager.go:414] "containerMap: removing container" podUID="72896412-b187-41b0-a942-ab1bad9f8fe9" containerName="test-curl-8743-a0b3da" Oct 30 13:14:39.414959 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:39.414912 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72896412-b187-41b0-a942-ab1bad9f8fe9" containerName="test-curl-8743-a0b3da" Oct 30 13:14:39.415028 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:39.415018 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="72896412-b187-41b0-a942-ab1bad9f8fe9" containerName="test-curl-8743-a0b3da" Oct 30 13:14:39.418679 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:39.418657 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" Oct 30 13:14:39.447129 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:39.447095 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8af0fe\"" Oct 30 13:14:39.451283 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:39.451250 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk"] Oct 30 13:14:39.477920 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:14:39.477877 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8743-a0b3da\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="72896412-b187-41b0-a942-ab1bad9f8fe9" pod="opendatahub/test-curl-8743-a0b3da" Oct 30 13:14:39.492121 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:39.492080 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72896412-b187-41b0-a942-ab1bad9f8fe9" path="/var/lib/kubelet/pods/72896412-b187-41b0-a942-ab1bad9f8fe9/volumes" Oct 30 13:14:39.621674 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:39.605846 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76blk\" (UniqueName: \"kubernetes.io/projected/58ed4375-0acf-48eb-97d3-2ccf5d1681f4-kube-api-access-76blk\") pod \"payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk\" (UID: \"58ed4375-0acf-48eb-97d3-2ccf5d1681f4\") " pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" Oct 30 13:14:39.637812 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:39.637768 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ed4375-0acf-48eb-97d3-2ccf5d1681f4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk\" (UID: \"58ed4375-0acf-48eb-97d3-2ccf5d1681f4\") " pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" Oct 30 13:14:39.766771 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:39.761352 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76blk\" (UniqueName: \"kubernetes.io/projected/58ed4375-0acf-48eb-97d3-2ccf5d1681f4-kube-api-access-76blk\") pod \"payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk\" (UID: \"58ed4375-0acf-48eb-97d3-2ccf5d1681f4\") " pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" Oct 30 13:14:39.766771 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:39.761684 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ed4375-0acf-48eb-97d3-2ccf5d1681f4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk\" (UID: \"58ed4375-0acf-48eb-97d3-2ccf5d1681f4\") " pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" Oct 30 13:14:39.766771 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:39.762909 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ed4375-0acf-48eb-97d3-2ccf5d1681f4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk\" (UID: \"58ed4375-0acf-48eb-97d3-2ccf5d1681f4\") " pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" Oct 30 13:14:39.791709 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:39.791677 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76blk\" (UniqueName: \"kubernetes.io/projected/58ed4375-0acf-48eb-97d3-2ccf5d1681f4-kube-api-access-76blk\") pod \"payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk\" (UID: \"58ed4375-0acf-48eb-97d3-2ccf5d1681f4\") " pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" Oct 30 13:14:40.032231 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:40.032138 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" Oct 30 13:14:40.348561 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:40.348522 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk"] Oct 30 13:14:40.350377 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:14:40.350346 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ed4375_0acf_48eb_97d3_2ccf5d1681f4.slice/crio-f73042d432c7a50ae640e124d9c02c74c97d24da76f2de69b378370d01ddff72 WatchSource:0}: Error finding container f73042d432c7a50ae640e124d9c02c74c97d24da76f2de69b378370d01ddff72: Status 404 returned error can't find the container with id f73042d432c7a50ae640e124d9c02c74c97d24da76f2de69b378370d01ddff72 Oct 30 13:14:40.995286 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:40.995240 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" event={"ID":"58ed4375-0acf-48eb-97d3-2ccf5d1681f4","Type":"ContainerStarted","Data":"f73042d432c7a50ae640e124d9c02c74c97d24da76f2de69b378370d01ddff72"} Oct 30 13:14:41.603931 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:41.603894 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg"] Oct 30 13:14:41.610631 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:41.610596 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:14:41.628005 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:41.627974 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg"] Oct 30 13:14:41.638076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:41.638026 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6b0881-serving-cert\"" Oct 30 13:14:41.703591 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:41.703557 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgp6\" (UniqueName: \"kubernetes.io/projected/1c26e7b5-7225-4121-aed4-332a5028b843-kube-api-access-5fgp6\") pod \"maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg\" (UID: \"1c26e7b5-7225-4121-aed4-332a5028b843\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:14:41.703823 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:41.703801 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c26e7b5-7225-4121-aed4-332a5028b843-metrics-certs\") pod \"maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg\" (UID: \"1c26e7b5-7225-4121-aed4-332a5028b843\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:14:41.703898 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:41.703857 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c26e7b5-7225-4121-aed4-332a5028b843-maas-api-tls\") pod \"maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg\" (UID: \"1c26e7b5-7225-4121-aed4-332a5028b843\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:14:41.704078 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:41.704054 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c26e7b5-7225-4121-aed4-332a5028b843-openshift-service-ca\") pod \"maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg\" (UID: \"1c26e7b5-7225-4121-aed4-332a5028b843\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:14:41.805423 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:41.805381 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgp6\" (UniqueName: \"kubernetes.io/projected/1c26e7b5-7225-4121-aed4-332a5028b843-kube-api-access-5fgp6\") pod \"maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg\" (UID: \"1c26e7b5-7225-4121-aed4-332a5028b843\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:14:41.805647 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:41.805592 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c26e7b5-7225-4121-aed4-332a5028b843-metrics-certs\") pod \"maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg\" (UID: \"1c26e7b5-7225-4121-aed4-332a5028b843\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:14:41.805647 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:41.805629 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c26e7b5-7225-4121-aed4-332a5028b843-maas-api-tls\") pod \"maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg\" (UID: \"1c26e7b5-7225-4121-aed4-332a5028b843\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:14:41.805779 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:41.805749 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c26e7b5-7225-4121-aed4-332a5028b843-openshift-service-ca\") pod \"maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg\" (UID: \"1c26e7b5-7225-4121-aed4-332a5028b843\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:14:41.806455 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:41.806430 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c26e7b5-7225-4121-aed4-332a5028b843-openshift-service-ca\") pod \"maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg\" (UID: \"1c26e7b5-7225-4121-aed4-332a5028b843\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:14:41.809902 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:41.809873 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c26e7b5-7225-4121-aed4-332a5028b843-metrics-certs\") pod \"maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg\" (UID: \"1c26e7b5-7225-4121-aed4-332a5028b843\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:14:41.810293 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:41.810270 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c26e7b5-7225-4121-aed4-332a5028b843-maas-api-tls\") pod \"maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg\" (UID: \"1c26e7b5-7225-4121-aed4-332a5028b843\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:14:41.831446 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:41.831404 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgp6\" (UniqueName: \"kubernetes.io/projected/1c26e7b5-7225-4121-aed4-332a5028b843-kube-api-access-5fgp6\") pod \"maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg\" (UID: \"1c26e7b5-7225-4121-aed4-332a5028b843\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:14:41.958692 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:41.958637 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:14:42.359964 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:42.359926 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg"] Oct 30 13:14:42.364935 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:14:42.364735 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c26e7b5_7225_4121_aed4_332a5028b843.slice/crio-f1f6d5da71e594b7593f186203286bdd7ca21ef4681c0a5a710629191b32df43 WatchSource:0}: Error finding container f1f6d5da71e594b7593f186203286bdd7ca21ef4681c0a5a710629191b32df43: Status 404 returned error can't find the container with id f1f6d5da71e594b7593f186203286bdd7ca21ef4681c0a5a710629191b32df43 Oct 30 13:14:43.110571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:43.104630 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" event={"ID":"58ed4375-0acf-48eb-97d3-2ccf5d1681f4","Type":"ContainerStarted","Data":"3dcc1795a4fa0e42eef3efba8d715efe88803a32f5ae51433a4e302d9d0a0e5f"} Oct 30 13:14:43.110571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:43.106088 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" Oct 30 13:14:43.124360 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:43.124322 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" event={"ID":"1c26e7b5-7225-4121-aed4-332a5028b843","Type":"ContainerStarted","Data":"ed917a83dd7ce4ae1f2d7b166330d639f948620788b2b570ef0ceb6e5e1beb6d"} Oct 30 13:14:43.124891 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:43.124868 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:14:43.125080 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:43.125061 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" event={"ID":"1c26e7b5-7225-4121-aed4-332a5028b843","Type":"ContainerStarted","Data":"f1f6d5da71e594b7593f186203286bdd7ca21ef4681c0a5a710629191b32df43"} Oct 30 13:14:43.169552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:43.166338 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" podStartSLOduration=1.693381031 podStartE2EDuration="4.166322456s" totalImagesPullingTime="2.472941425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:43.161513551 +0000 UTC m=+1339.793700722" watchObservedRunningTime="2026-10-30 13:14:43.166322456 +0000 UTC m=+1339.798509651" Oct 30 13:14:43.213784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:43.213713 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" podStartSLOduration=1.833568015 podStartE2EDuration="2.213692743s" totalImagesPullingTime="380.124728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:43.211902174 +0000 UTC m=+1339.844089349" watchObservedRunningTime="2026-10-30 13:14:43.213692743 +0000 UTC m=+1339.845879907" Oct 30 13:14:49.137997 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:49.137965 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:14:55.143784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:14:55.143753 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" Oct 30 13:14:59.401397 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:14:59.400999 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:15:00.346086 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.346049 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn"] Oct 30 13:15:00.351432 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.351408 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp"] Oct 30 13:15:00.351620 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.351550 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" Oct 30 13:15:00.356868 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.356725 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6"] Oct 30 13:15:00.357013 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.356871 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" Oct 30 13:15:00.361972 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.361607 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh"] Oct 30 13:15:00.361972 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.361694 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" Oct 30 13:15:00.367891 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.367864 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" Oct 30 13:15:00.417060 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.416468 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn"] Oct 30 13:15:00.423067 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.421604 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:15:00.429194 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.429124 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp"] Oct 30 13:15:00.437480 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.437418 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh"] Oct 30 13:15:00.441592 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.441561 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8djxx\" (UniqueName: \"kubernetes.io/projected/3e792233-9490-4134-acce-5584ab43123a-kube-api-access-8djxx\") pod \"maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn\" (UID: \"3e792233-9490-4134-acce-5584ab43123a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" Oct 30 13:15:00.441730 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.441693 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hztm\" (UniqueName: \"kubernetes.io/projected/a0aace59-a54a-447a-ba91-3ab588b2c2fe-kube-api-access-5hztm\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6\" (UID: \"a0aace59-a54a-447a-ba91-3ab588b2c2fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" Oct 30 13:15:00.441937 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.441907 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d86f05b-db3f-417f-b5ab-395ebcbb4a0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp\" (UID: \"9d86f05b-db3f-417f-b5ab-395ebcbb4a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" Oct 30 13:15:00.442173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.442152 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ede3d71-cc65-4660-b836-6f05f27ea6cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889435-dlkxh\" (UID: \"9ede3d71-cc65-4660-b836-6f05f27ea6cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" Oct 30 13:15:00.442327 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.442311 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9pcq\" (UniqueName: \"kubernetes.io/projected/9ede3d71-cc65-4660-b836-6f05f27ea6cc-kube-api-access-j9pcq\") pod \"maas-api-key-cleanup-29889435-dlkxh\" (UID: \"9ede3d71-cc65-4660-b836-6f05f27ea6cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" Oct 30 13:15:00.442546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.442525 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0aace59-a54a-447a-ba91-3ab588b2c2fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6\" (UID: \"a0aace59-a54a-447a-ba91-3ab588b2c2fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" Oct 30 13:15:00.442770 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.442751 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6db5q\" (UniqueName: \"kubernetes.io/projected/9d86f05b-db3f-417f-b5ab-395ebcbb4a0e-kube-api-access-6db5q\") pod \"maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp\" (UID: \"9d86f05b-db3f-417f-b5ab-395ebcbb4a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" Oct 30 13:15:00.442962 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.442945 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e792233-9490-4134-acce-5584ab43123a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn\" (UID: \"3e792233-9490-4134-acce-5584ab43123a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" Oct 30 13:15:00.458979 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.458261 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6"] Oct 30 13:15:00.547876 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.547830 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djxx\" (UniqueName: \"kubernetes.io/projected/3e792233-9490-4134-acce-5584ab43123a-kube-api-access-8djxx\") pod \"maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn\" (UID: \"3e792233-9490-4134-acce-5584ab43123a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" Oct 30 13:15:00.548075 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.547971 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hztm\" (UniqueName: \"kubernetes.io/projected/a0aace59-a54a-447a-ba91-3ab588b2c2fe-kube-api-access-5hztm\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6\" (UID: \"a0aace59-a54a-447a-ba91-3ab588b2c2fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" Oct 30 13:15:00.548153 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.548135 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d86f05b-db3f-417f-b5ab-395ebcbb4a0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp\" (UID: \"9d86f05b-db3f-417f-b5ab-395ebcbb4a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" Oct 30 13:15:00.548219 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.548197 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ede3d71-cc65-4660-b836-6f05f27ea6cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889435-dlkxh\" (UID: \"9ede3d71-cc65-4660-b836-6f05f27ea6cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" Oct 30 13:15:00.548277 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.548226 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9pcq\" (UniqueName: \"kubernetes.io/projected/9ede3d71-cc65-4660-b836-6f05f27ea6cc-kube-api-access-j9pcq\") pod \"maas-api-key-cleanup-29889435-dlkxh\" (UID: \"9ede3d71-cc65-4660-b836-6f05f27ea6cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" Oct 30 13:15:00.548337 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.548278 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0aace59-a54a-447a-ba91-3ab588b2c2fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6\" (UID: \"a0aace59-a54a-447a-ba91-3ab588b2c2fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" Oct 30 13:15:00.548394 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.548348 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6db5q\" (UniqueName: \"kubernetes.io/projected/9d86f05b-db3f-417f-b5ab-395ebcbb4a0e-kube-api-access-6db5q\") pod \"maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp\" (UID: \"9d86f05b-db3f-417f-b5ab-395ebcbb4a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" Oct 30 13:15:00.548449 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.548406 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e792233-9490-4134-acce-5584ab43123a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn\" (UID: \"3e792233-9490-4134-acce-5584ab43123a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" Oct 30 13:15:00.550545 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.549234 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0aace59-a54a-447a-ba91-3ab588b2c2fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6\" (UID: \"a0aace59-a54a-447a-ba91-3ab588b2c2fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" Oct 30 13:15:00.550545 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.549321 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e792233-9490-4134-acce-5584ab43123a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn\" (UID: \"3e792233-9490-4134-acce-5584ab43123a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" Oct 30 13:15:00.550545 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.549375 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ede3d71-cc65-4660-b836-6f05f27ea6cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889435-dlkxh\" (UID: \"9ede3d71-cc65-4660-b836-6f05f27ea6cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" Oct 30 13:15:00.550545 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.550171 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d86f05b-db3f-417f-b5ab-395ebcbb4a0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp\" (UID: \"9d86f05b-db3f-417f-b5ab-395ebcbb4a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" Oct 30 13:15:00.598574 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.594978 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hztm\" (UniqueName: \"kubernetes.io/projected/a0aace59-a54a-447a-ba91-3ab588b2c2fe-kube-api-access-5hztm\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6\" (UID: \"a0aace59-a54a-447a-ba91-3ab588b2c2fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" Oct 30 13:15:00.598574 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.597190 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9pcq\" (UniqueName: \"kubernetes.io/projected/9ede3d71-cc65-4660-b836-6f05f27ea6cc-kube-api-access-j9pcq\") pod \"maas-api-key-cleanup-29889435-dlkxh\" (UID: \"9ede3d71-cc65-4660-b836-6f05f27ea6cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" Oct 30 13:15:00.599995 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.599964 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6db5q\" (UniqueName: \"kubernetes.io/projected/9d86f05b-db3f-417f-b5ab-395ebcbb4a0e-kube-api-access-6db5q\") pod \"maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp\" (UID: \"9d86f05b-db3f-417f-b5ab-395ebcbb4a0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" Oct 30 13:15:00.604372 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.604313 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8djxx\" (UniqueName: \"kubernetes.io/projected/3e792233-9490-4134-acce-5584ab43123a-kube-api-access-8djxx\") pod \"maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn\" (UID: \"3e792233-9490-4134-acce-5584ab43123a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" Oct 30 13:15:00.692606 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.692568 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" Oct 30 13:15:00.733112 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.733072 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" Oct 30 13:15:00.751175 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.751148 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" Oct 30 13:15:00.767199 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:00.767169 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" Oct 30 13:15:01.039192 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:01.039161 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn"] Oct 30 13:15:01.039578 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:15:01.039552 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice/crio-2a4854682bc850060336bd6497bf4f91fa6dce2a90552b13d418c0f0852fe925 WatchSource:0}: Error finding container 2a4854682bc850060336bd6497bf4f91fa6dce2a90552b13d418c0f0852fe925: Status 404 returned error can't find the container with id 2a4854682bc850060336bd6497bf4f91fa6dce2a90552b13d418c0f0852fe925 Oct 30 13:15:01.149539 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:01.149511 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh"] Oct 30 13:15:01.150170 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:15:01.150137 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ede3d71_cc65_4660_b836_6f05f27ea6cc.slice/crio-d22c50035991cd3ca72623ca149d4480e8d10102a8ee56f2801025104110a23f WatchSource:0}: Error finding container d22c50035991cd3ca72623ca149d4480e8d10102a8ee56f2801025104110a23f: Status 404 returned error can't find the container with id d22c50035991cd3ca72623ca149d4480e8d10102a8ee56f2801025104110a23f Oct 30 13:15:01.156209 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:15:01.156154 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d86f05b_db3f_417f_b5ab_395ebcbb4a0e.slice/crio-f00057a033dcbec26b5c936bbcff2cb87abbd3fa34c95898b6a84acc7bce7a46 WatchSource:0}: Error finding container f00057a033dcbec26b5c936bbcff2cb87abbd3fa34c95898b6a84acc7bce7a46: Status 404 returned error can't find the container with id f00057a033dcbec26b5c936bbcff2cb87abbd3fa34c95898b6a84acc7bce7a46 Oct 30 13:15:01.156351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:01.156335 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp"] Oct 30 13:15:01.178084 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:15:01.178060 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0aace59_a54a_447a_ba91_3ab588b2c2fe.slice/crio-383664d230b0a28a1d6b7ef26a4d29da7e0745dbe3528bd2f5b8eea6183cc5d8 WatchSource:0}: Error finding container 383664d230b0a28a1d6b7ef26a4d29da7e0745dbe3528bd2f5b8eea6183cc5d8: Status 404 returned error can't find the container with id 383664d230b0a28a1d6b7ef26a4d29da7e0745dbe3528bd2f5b8eea6183cc5d8 Oct 30 13:15:01.178476 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:01.178448 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6"] Oct 30 13:15:01.279417 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:01.279381 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" event={"ID":"a0aace59-a54a-447a-ba91-3ab588b2c2fe","Type":"ContainerStarted","Data":"383664d230b0a28a1d6b7ef26a4d29da7e0745dbe3528bd2f5b8eea6183cc5d8"} Oct 30 13:15:01.280897 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:01.280868 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" event={"ID":"9ede3d71-cc65-4660-b836-6f05f27ea6cc","Type":"ContainerStarted","Data":"d22c50035991cd3ca72623ca149d4480e8d10102a8ee56f2801025104110a23f"} Oct 30 13:15:01.282064 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:01.282041 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" event={"ID":"9d86f05b-db3f-417f-b5ab-395ebcbb4a0e","Type":"ContainerStarted","Data":"f00057a033dcbec26b5c936bbcff2cb87abbd3fa34c95898b6a84acc7bce7a46"} Oct 30 13:15:01.283241 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:01.283218 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" event={"ID":"3e792233-9490-4134-acce-5584ab43123a","Type":"ContainerStarted","Data":"2a4854682bc850060336bd6497bf4f91fa6dce2a90552b13d418c0f0852fe925"} Oct 30 13:15:08.571593 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:08.571253 7010 generic.go:356] "Generic (PLEG): container finished" podID="a0aace59-a54a-447a-ba91-3ab588b2c2fe" containerID="0081c6db1fbd9a900a20a6943f6d620cc468e0911a9f820aebb13bc0ebc72a87" exitCode=60 Oct 30 13:15:08.571593 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:08.571286 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" event={"ID":"a0aace59-a54a-447a-ba91-3ab588b2c2fe","Type":"ContainerDied","Data":"0081c6db1fbd9a900a20a6943f6d620cc468e0911a9f820aebb13bc0ebc72a87"} Oct 30 13:15:08.572185 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:08.571900 7010 scope.go:122] "RemoveContainer" containerID="0081c6db1fbd9a900a20a6943f6d620cc468e0911a9f820aebb13bc0ebc72a87" Oct 30 13:15:08.574762 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:08.574724 7010 generic.go:356] "Generic (PLEG): container finished" podID="9ede3d71-cc65-4660-b836-6f05f27ea6cc" containerID="fc48a044f3a9ca999819da23653b2e0965632f8620342d8947e6cb35e330bb98" exitCode=60 Oct 30 13:15:08.574935 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:08.574819 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" event={"ID":"9ede3d71-cc65-4660-b836-6f05f27ea6cc","Type":"ContainerDied","Data":"fc48a044f3a9ca999819da23653b2e0965632f8620342d8947e6cb35e330bb98"} Oct 30 13:15:08.575344 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:08.575325 7010 scope.go:122] "RemoveContainer" containerID="fc48a044f3a9ca999819da23653b2e0965632f8620342d8947e6cb35e330bb98" Oct 30 13:15:08.588775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:08.588746 7010 generic.go:356] "Generic (PLEG): container finished" podID="9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" containerID="5766b65a421c6dd9a2d656402745246b6b551b16915dd0294dfbdb31b4be12a8" exitCode=60 Oct 30 13:15:08.588923 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:08.588838 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" event={"ID":"9d86f05b-db3f-417f-b5ab-395ebcbb4a0e","Type":"ContainerDied","Data":"5766b65a421c6dd9a2d656402745246b6b551b16915dd0294dfbdb31b4be12a8"} Oct 30 13:15:08.589372 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:08.589337 7010 scope.go:122] "RemoveContainer" containerID="5766b65a421c6dd9a2d656402745246b6b551b16915dd0294dfbdb31b4be12a8" Oct 30 13:15:08.593008 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:08.592980 7010 generic.go:356] "Generic (PLEG): container finished" podID="3e792233-9490-4134-acce-5584ab43123a" containerID="5d187774440ac65f20ae828461e02ca0cf87f0b99cb7591083b1611dd3855318" exitCode=60 Oct 30 13:15:08.593102 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:08.593057 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" event={"ID":"3e792233-9490-4134-acce-5584ab43123a","Type":"ContainerDied","Data":"5d187774440ac65f20ae828461e02ca0cf87f0b99cb7591083b1611dd3855318"} Oct 30 13:15:08.593537 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:08.593514 7010 scope.go:122] "RemoveContainer" containerID="5d187774440ac65f20ae828461e02ca0cf87f0b99cb7591083b1611dd3855318" Oct 30 13:15:09.604431 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:09.604394 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" event={"ID":"9ede3d71-cc65-4660-b836-6f05f27ea6cc","Type":"ContainerStarted","Data":"4ad3c784013b08b56efaca0e198a74d3da828c2905bc544e999ca60a8bc364fb"} Oct 30 13:15:09.607406 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:09.607372 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" event={"ID":"9d86f05b-db3f-417f-b5ab-395ebcbb4a0e","Type":"ContainerStarted","Data":"771238082f9b17f025280b04036f194da6c253099a8c9bdca5f94d04378c5aee"} Oct 30 13:15:09.609554 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:09.609530 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" event={"ID":"3e792233-9490-4134-acce-5584ab43123a","Type":"ContainerStarted","Data":"037956ecb997a02ec2dc93097ba8bd38b2122e10a5faef5b5830454ce5698443"} Oct 30 13:15:09.611668 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:09.611644 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" event={"ID":"a0aace59-a54a-447a-ba91-3ab588b2c2fe","Type":"ContainerStarted","Data":"445c7374db4ddd7eb38794a211790c198fca9d16592bb0759d1d5debde8dee43"} Oct 30 13:15:09.645726 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:09.645575 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" podStartSLOduration=3.752351262 podStartE2EDuration="9.6455549s" totalImagesPullingTime="5.893203638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:09.644223753 +0000 UTC m=+1366.276410914" watchObservedRunningTime="2026-10-30 13:15:09.6455549 +0000 UTC m=+1366.277742066" Oct 30 13:15:09.694550 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:09.694471 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" podStartSLOduration=3.805778539 podStartE2EDuration="9.694456714s" totalImagesPullingTime="5.888678175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:09.692395598 +0000 UTC m=+1366.324582786" watchObservedRunningTime="2026-10-30 13:15:09.694456714 +0000 UTC m=+1366.326643876" Oct 30 13:15:09.737695 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:09.737643 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" podStartSLOduration=3.788376629 podStartE2EDuration="9.737628702s" totalImagesPullingTime="5.949252073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:09.736004371 +0000 UTC m=+1366.368191533" watchObservedRunningTime="2026-10-30 13:15:09.737628702 +0000 UTC m=+1366.369815864" Oct 30 13:15:09.799871 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:09.799804 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" podStartSLOduration=3.912051182 podStartE2EDuration="9.799788281s" totalImagesPullingTime="5.887737099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:09.798029834 +0000 UTC m=+1366.430216996" watchObservedRunningTime="2026-10-30 13:15:09.799788281 +0000 UTC m=+1366.431975443" Oct 30 13:15:09.900044 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:09.900003 7010 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice/crio-037956ecb997a02ec2dc93097ba8bd38b2122e10a5faef5b5830454ce5698443.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:15:10.622359 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:10.622323 7010 generic.go:356] "Generic (PLEG): container finished" podID="9ede3d71-cc65-4660-b836-6f05f27ea6cc" containerID="4ad3c784013b08b56efaca0e198a74d3da828c2905bc544e999ca60a8bc364fb" exitCode=60 Oct 30 13:15:10.622870 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:10.622411 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" event={"ID":"9ede3d71-cc65-4660-b836-6f05f27ea6cc","Type":"ContainerDied","Data":"4ad3c784013b08b56efaca0e198a74d3da828c2905bc544e999ca60a8bc364fb"} Oct 30 13:15:10.622870 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:10.622478 7010 scope.go:122] "RemoveContainer" containerID="fc48a044f3a9ca999819da23653b2e0965632f8620342d8947e6cb35e330bb98" Oct 30 13:15:10.622983 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:10.622872 7010 scope.go:122] "RemoveContainer" containerID="4ad3c784013b08b56efaca0e198a74d3da828c2905bc544e999ca60a8bc364fb" Oct 30 13:15:10.623360 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:10.623333 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889435-dlkxh_odh-ai-gateway-infra(9ede3d71-cc65-4660-b836-6f05f27ea6cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" podUID="9ede3d71-cc65-4660-b836-6f05f27ea6cc" Oct 30 13:15:10.624720 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:10.624696 7010 generic.go:356] "Generic (PLEG): container finished" podID="9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" containerID="771238082f9b17f025280b04036f194da6c253099a8c9bdca5f94d04378c5aee" exitCode=60 Oct 30 13:15:10.624813 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:10.624781 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" event={"ID":"9d86f05b-db3f-417f-b5ab-395ebcbb4a0e","Type":"ContainerDied","Data":"771238082f9b17f025280b04036f194da6c253099a8c9bdca5f94d04378c5aee"} Oct 30 13:15:10.625086 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:10.625064 7010 scope.go:122] "RemoveContainer" containerID="771238082f9b17f025280b04036f194da6c253099a8c9bdca5f94d04378c5aee" Oct 30 13:15:10.625532 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:10.625472 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp_odh-ai-gateway-infra(9d86f05b-db3f-417f-b5ab-395ebcbb4a0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" podUID="9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" Oct 30 13:15:10.626900 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:10.626879 7010 generic.go:356] "Generic (PLEG): container finished" podID="3e792233-9490-4134-acce-5584ab43123a" containerID="037956ecb997a02ec2dc93097ba8bd38b2122e10a5faef5b5830454ce5698443" exitCode=60 Oct 30 13:15:10.627013 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:10.626963 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" event={"ID":"3e792233-9490-4134-acce-5584ab43123a","Type":"ContainerDied","Data":"037956ecb997a02ec2dc93097ba8bd38b2122e10a5faef5b5830454ce5698443"} Oct 30 13:15:10.627252 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:10.627235 7010 scope.go:122] "RemoveContainer" containerID="037956ecb997a02ec2dc93097ba8bd38b2122e10a5faef5b5830454ce5698443" Oct 30 13:15:10.627623 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:10.627599 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn_odh-ai-gateway-infra(3e792233-9490-4134-acce-5584ab43123a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" podUID="3e792233-9490-4134-acce-5584ab43123a" Oct 30 13:15:10.628970 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:10.628950 7010 generic.go:356] "Generic (PLEG): container finished" podID="a0aace59-a54a-447a-ba91-3ab588b2c2fe" containerID="445c7374db4ddd7eb38794a211790c198fca9d16592bb0759d1d5debde8dee43" exitCode=60 Oct 30 13:15:10.629066 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:10.628974 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" event={"ID":"a0aace59-a54a-447a-ba91-3ab588b2c2fe","Type":"ContainerDied","Data":"445c7374db4ddd7eb38794a211790c198fca9d16592bb0759d1d5debde8dee43"} Oct 30 13:15:10.629281 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:10.629265 7010 scope.go:122] "RemoveContainer" containerID="445c7374db4ddd7eb38794a211790c198fca9d16592bb0759d1d5debde8dee43" Oct 30 13:15:10.629653 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:10.629626 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6_odh-ai-gateway-infra(a0aace59-a54a-447a-ba91-3ab588b2c2fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" podUID="a0aace59-a54a-447a-ba91-3ab588b2c2fe" Oct 30 13:15:10.637862 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:10.637841 7010 scope.go:122] "RemoveContainer" containerID="5766b65a421c6dd9a2d656402745246b6b551b16915dd0294dfbdb31b4be12a8" Oct 30 13:15:10.652019 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:10.651994 7010 scope.go:122] "RemoveContainer" containerID="5d187774440ac65f20ae828461e02ca0cf87f0b99cb7591083b1611dd3855318" Oct 30 13:15:10.667427 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:10.667400 7010 scope.go:122] "RemoveContainer" containerID="0081c6db1fbd9a900a20a6943f6d620cc468e0911a9f820aebb13bc0ebc72a87" Oct 30 13:15:11.642621 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:11.640672 7010 scope.go:122] "RemoveContainer" containerID="4ad3c784013b08b56efaca0e198a74d3da828c2905bc544e999ca60a8bc364fb" Oct 30 13:15:11.642621 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:11.641173 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889435-dlkxh_odh-ai-gateway-infra(9ede3d71-cc65-4660-b836-6f05f27ea6cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" podUID="9ede3d71-cc65-4660-b836-6f05f27ea6cc" Oct 30 13:15:11.643544 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:11.643473 7010 scope.go:122] "RemoveContainer" containerID="771238082f9b17f025280b04036f194da6c253099a8c9bdca5f94d04378c5aee" Oct 30 13:15:11.643943 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:11.643919 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp_odh-ai-gateway-infra(9d86f05b-db3f-417f-b5ab-395ebcbb4a0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" podUID="9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" Oct 30 13:15:11.645605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:11.645585 7010 scope.go:122] "RemoveContainer" containerID="037956ecb997a02ec2dc93097ba8bd38b2122e10a5faef5b5830454ce5698443" Oct 30 13:15:11.645967 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:11.645936 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn_odh-ai-gateway-infra(3e792233-9490-4134-acce-5584ab43123a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" podUID="3e792233-9490-4134-acce-5584ab43123a" Oct 30 13:15:11.647610 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:11.647592 7010 scope.go:122] "RemoveContainer" containerID="445c7374db4ddd7eb38794a211790c198fca9d16592bb0759d1d5debde8dee43" Oct 30 13:15:11.647930 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:11.647909 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6_odh-ai-gateway-infra(a0aace59-a54a-447a-ba91-3ab588b2c2fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" podUID="a0aace59-a54a-447a-ba91-3ab588b2c2fe" Oct 30 13:15:12.656809 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:12.656777 7010 generic.go:356] "Generic (PLEG): container finished" podID="0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" containerID="2b2e23a440240fe691fedecbf6d296595183aa9701365b04891203b93b90f843" exitCode=0 Oct 30 13:15:12.657261 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:12.656818 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" event={"ID":"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb","Type":"ContainerDied","Data":"2b2e23a440240fe691fedecbf6d296595183aa9701365b04891203b93b90f843"} Oct 30 13:15:12.657331 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:12.657263 7010 scope.go:122] "RemoveContainer" containerID="2b2e23a440240fe691fedecbf6d296595183aa9701365b04891203b93b90f843" Oct 30 13:15:12.659189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:12.659156 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:15:12.659279 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:12.659241 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:15:12.659279 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:12.659274 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:15:13.272261 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:13.272218 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:15:13.666203 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:13.666170 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" event={"ID":"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb","Type":"ContainerStarted","Data":"157cff6432a23db88530dd65157cf34b85d685cee2310982c216560a2e95a1d0"} Oct 30 13:15:14.339183 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:14.339151 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:15:14.340440 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:14.340407 7010 patch_prober.go:29] interesting pod/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:15:14.340616 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:14.340457 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" podUID="0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 13:15:15.342317 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:15.341865 7010 patch_prober.go:29] interesting pod/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:15:15.342317 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:15.341928 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" podUID="0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 13:15:16.339627 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:16.339584 7010 patch_prober.go:29] interesting pod/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:15:16.339812 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:16.339657 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" podUID="0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 13:15:17.338722 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:17.338680 7010 patch_prober.go:29] interesting pod/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:15:17.339237 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:17.338767 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" podUID="0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 13:15:18.339167 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:18.339124 7010 patch_prober.go:29] interesting pod/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 13:15:18.339711 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:18.339195 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" podUID="0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 13:15:19.343867 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:19.343832 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:15:19.344264 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:19.344234 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:15:19.344953 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:19.344933 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:15:20.481983 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:20.481950 7010 scope.go:122] "RemoveContainer" containerID="037956ecb997a02ec2dc93097ba8bd38b2122e10a5faef5b5830454ce5698443" Oct 30 13:15:21.494697 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:21.494662 7010 scope.go:122] "RemoveContainer" containerID="445c7374db4ddd7eb38794a211790c198fca9d16592bb0759d1d5debde8dee43" Oct 30 13:15:21.774119 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:21.774036 7010 generic.go:356] "Generic (PLEG): container finished" podID="3e792233-9490-4134-acce-5584ab43123a" containerID="4ebce022764a53b0aaa2e3ac751df1671ff6e6dd1924574a20eb5115b609e0a6" exitCode=60 Oct 30 13:15:21.774119 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:21.774107 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" event={"ID":"3e792233-9490-4134-acce-5584ab43123a","Type":"ContainerDied","Data":"4ebce022764a53b0aaa2e3ac751df1671ff6e6dd1924574a20eb5115b609e0a6"} Oct 30 13:15:21.774333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:21.774169 7010 scope.go:122] "RemoveContainer" containerID="037956ecb997a02ec2dc93097ba8bd38b2122e10a5faef5b5830454ce5698443" Oct 30 13:15:21.774633 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:21.774615 7010 scope.go:122] "RemoveContainer" containerID="4ebce022764a53b0aaa2e3ac751df1671ff6e6dd1924574a20eb5115b609e0a6" Oct 30 13:15:21.775051 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:21.775023 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn_odh-ai-gateway-infra(3e792233-9490-4134-acce-5584ab43123a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" podUID="3e792233-9490-4134-acce-5584ab43123a" Oct 30 13:15:22.784160 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:22.784126 7010 generic.go:356] "Generic (PLEG): container finished" podID="a0aace59-a54a-447a-ba91-3ab588b2c2fe" containerID="64706c1906454d880171a078a347aaf750dc8bab675c101992dd42f36fd76ec9" exitCode=60 Oct 30 13:15:22.784604 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:22.784199 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" event={"ID":"a0aace59-a54a-447a-ba91-3ab588b2c2fe","Type":"ContainerDied","Data":"64706c1906454d880171a078a347aaf750dc8bab675c101992dd42f36fd76ec9"} Oct 30 13:15:22.784604 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:22.784255 7010 scope.go:122] "RemoveContainer" containerID="445c7374db4ddd7eb38794a211790c198fca9d16592bb0759d1d5debde8dee43" Oct 30 13:15:22.784945 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:22.784799 7010 scope.go:122] "RemoveContainer" containerID="64706c1906454d880171a078a347aaf750dc8bab675c101992dd42f36fd76ec9" Oct 30 13:15:22.785591 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:22.785290 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6_odh-ai-gateway-infra(a0aace59-a54a-447a-ba91-3ab588b2c2fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" podUID="a0aace59-a54a-447a-ba91-3ab588b2c2fe" Oct 30 13:15:22.829775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:22.829738 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn"] Oct 30 13:15:23.009692 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.009666 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" Oct 30 13:15:23.017671 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.017641 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz"] Oct 30 13:15:23.017842 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.017819 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" podUID="5d4b02d9-cc6d-4783-8c2f-c9c39215e18c" containerName="maas-api" containerID="cri-o://c5995f589c0a666b8dc9dcd79862719add86f32442768980518654bb5c6ba77e" gracePeriod=30 Oct 30 13:15:23.120763 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.120720 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e792233-9490-4134-acce-5584ab43123a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e792233-9490-4134-acce-5584ab43123a-maas-api-service-ca\") pod \"3e792233-9490-4134-acce-5584ab43123a\" (UID: \"3e792233-9490-4134-acce-5584ab43123a\") " Oct 30 13:15:23.120951 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.120932 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e792233-9490-4134-acce-5584ab43123a-kube-api-access-8djxx\" (UniqueName: \"kubernetes.io/projected/3e792233-9490-4134-acce-5584ab43123a-kube-api-access-8djxx\") pod \"3e792233-9490-4134-acce-5584ab43123a\" (UID: \"3e792233-9490-4134-acce-5584ab43123a\") " Oct 30 13:15:23.121373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.121343 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e792233-9490-4134-acce-5584ab43123a-maas-api-service-ca" pod "3e792233-9490-4134-acce-5584ab43123a" (UID: "3e792233-9490-4134-acce-5584ab43123a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:23.122374 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.122348 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e792233-9490-4134-acce-5584ab43123a-maas-api-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:23.124506 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.124467 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e792233-9490-4134-acce-5584ab43123a-kube-api-access-8djxx" pod "3e792233-9490-4134-acce-5584ab43123a" (UID: "3e792233-9490-4134-acce-5584ab43123a"). InnerVolumeSpecName "kube-api-access-8djxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:23.224533 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.224415 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8djxx\" (UniqueName: \"kubernetes.io/projected/3e792233-9490-4134-acce-5584ab43123a-kube-api-access-8djxx\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:23.565513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.561853 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:15:23.674919 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.674067 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-kube-api-access-f52v7\" (UniqueName: \"kubernetes.io/projected/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-kube-api-access-f52v7\") pod \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " Oct 30 13:15:23.674919 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.674227 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-metrics-certs\") pod \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " Oct 30 13:15:23.674919 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.674311 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-openshift-service-ca\") pod \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " Oct 30 13:15:23.674919 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.674544 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-maas-api-tls\") pod \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " Oct 30 13:15:23.718733 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.714239 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-openshift-service-ca" pod "5d4b02d9-cc6d-4783-8c2f-c9c39215e18c" (UID: "5d4b02d9-cc6d-4783-8c2f-c9c39215e18c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:23.746525 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.743674 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-kube-api-access-f52v7" pod "5d4b02d9-cc6d-4783-8c2f-c9c39215e18c" (UID: "5d4b02d9-cc6d-4783-8c2f-c9c39215e18c"). InnerVolumeSpecName "kube-api-access-f52v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:23.750508 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.746993 7010 scope.go:122] "RemoveContainer" containerID="771238082f9b17f025280b04036f194da6c253099a8c9bdca5f94d04378c5aee" Oct 30 13:15:23.786544 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.782536 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f52v7\" (UniqueName: \"kubernetes.io/projected/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-kube-api-access-f52v7\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:23.786544 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.782570 7010 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-openshift-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:23.867757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.845939 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp"] Oct 30 13:15:23.867757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.847100 7010 cpu_manager.go:414] "containerMap: removing container" podUID="3e792233-9490-4134-acce-5584ab43123a" containerName="cleanup" Oct 30 13:15:23.867757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.847120 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e792233-9490-4134-acce-5584ab43123a" containerName="cleanup" Oct 30 13:15:23.867757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.847138 7010 cpu_manager.go:414] "containerMap: removing container" podUID="3e792233-9490-4134-acce-5584ab43123a" containerName="cleanup" Oct 30 13:15:23.867757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.847148 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e792233-9490-4134-acce-5584ab43123a" containerName="cleanup" Oct 30 13:15:23.867757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.847170 7010 cpu_manager.go:414] "containerMap: removing container" podUID="3e792233-9490-4134-acce-5584ab43123a" containerName="cleanup" Oct 30 13:15:23.867757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.847177 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e792233-9490-4134-acce-5584ab43123a" containerName="cleanup" Oct 30 13:15:23.867757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.847226 7010 cpu_manager.go:414] "containerMap: removing container" podUID="5d4b02d9-cc6d-4783-8c2f-c9c39215e18c" containerName="maas-api" Oct 30 13:15:23.867757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.847233 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d4b02d9-cc6d-4783-8c2f-c9c39215e18c" containerName="maas-api" Oct 30 13:15:23.867757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.847383 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d4b02d9-cc6d-4783-8c2f-c9c39215e18c" containerName="maas-api" Oct 30 13:15:23.867757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.847406 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e792233-9490-4134-acce-5584ab43123a" containerName="cleanup" Oct 30 13:15:23.867757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.847414 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e792233-9490-4134-acce-5584ab43123a" containerName="cleanup" Oct 30 13:15:23.867757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.847427 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e792233-9490-4134-acce-5584ab43123a" containerName="cleanup" Oct 30 13:15:23.867757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.854947 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:15:23.867757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.866125 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" event={"ID":"3e792233-9490-4134-acce-5584ab43123a","Type":"ContainerDied","Data":"2a4854682bc850060336bd6497bf4f91fa6dce2a90552b13d418c0f0852fe925"} Oct 30 13:15:23.867757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.866197 7010 scope.go:122] "RemoveContainer" containerID="4ebce022764a53b0aaa2e3ac751df1671ff6e6dd1924574a20eb5115b609e0a6" Oct 30 13:15:23.867757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.866284 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn" Oct 30 13:15:23.873099 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.873064 7010 generic.go:356] "Generic (PLEG): container finished" podID="5d4b02d9-cc6d-4783-8c2f-c9c39215e18c" containerID="c5995f589c0a666b8dc9dcd79862719add86f32442768980518654bb5c6ba77e" exitCode=0 Oct 30 13:15:23.873519 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.873475 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" event={"ID":"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c","Type":"ContainerDied","Data":"c5995f589c0a666b8dc9dcd79862719add86f32442768980518654bb5c6ba77e"} Oct 30 13:15:23.873709 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.873685 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" event={"ID":"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c","Type":"ContainerDied","Data":"6e0fe9699646b9388f6989923513bd84df3817af91b4485077b9f119acf9cde2"} Oct 30 13:15:23.873913 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.873898 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz" Oct 30 13:15:23.889375 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.889316 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-metrics-certs" pod "5d4b02d9-cc6d-4783-8c2f-c9c39215e18c" (UID: "5d4b02d9-cc6d-4783-8c2f-c9c39215e18c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:15:23.890054 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.889587 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-metrics-certs\") pod \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\" (UID: \"5d4b02d9-cc6d-4783-8c2f-c9c39215e18c\") " Oct 30 13:15:23.890261 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:15:23.890217 7010 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c/volumes/kubernetes.io~secret/metrics-certs Oct 30 13:15:23.890261 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.890237 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-metrics-certs" pod "5d4b02d9-cc6d-4783-8c2f-c9c39215e18c" (UID: "5d4b02d9-cc6d-4783-8c2f-c9c39215e18c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:15:23.891035 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.891000 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-maas-api-tls" pod "5d4b02d9-cc6d-4783-8c2f-c9c39215e18c" (UID: "5d4b02d9-cc6d-4783-8c2f-c9c39215e18c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:15:23.918921 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.913322 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-9c6033-serving-cert\"" Oct 30 13:15:23.918921 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.915726 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp"] Oct 30 13:15:23.924105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.919557 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r996z\" (UniqueName: \"kubernetes.io/projected/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-kube-api-access-r996z\") pod \"maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp\" (UID: \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:15:23.924105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.919740 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp\" (UID: \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:15:23.924105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.919907 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp\" (UID: \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:15:23.924105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.919976 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp\" (UID: \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:15:23.924105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.920217 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-maas-api-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:23.924105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.920232 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:23.943704 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:23.943662 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6"] Oct 30 13:15:24.040801 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.030975 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r996z\" (UniqueName: \"kubernetes.io/projected/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-kube-api-access-r996z\") pod \"maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp\" (UID: \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:15:24.040801 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.031126 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp\" (UID: \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:15:24.040801 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.031272 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp\" (UID: \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:15:24.040801 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.031331 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp\" (UID: \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:15:24.040801 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.032273 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp\" (UID: \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:15:24.124937 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.099537 7010 scope.go:122] "RemoveContainer" containerID="c5995f589c0a666b8dc9dcd79862719add86f32442768980518654bb5c6ba77e" Oct 30 13:15:24.124937 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.113254 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp\" (UID: \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:15:24.124937 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.120148 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp\" (UID: \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:15:24.155527 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.149039 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r996z\" (UniqueName: \"kubernetes.io/projected/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-kube-api-access-r996z\") pod \"maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp\" (UID: \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:15:24.183536 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.179622 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn"] Oct 30 13:15:24.207529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.207468 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3778dacd-29889435-fbxfn"] Oct 30 13:15:24.210030 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.209997 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:15:24.271690 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.261788 7010 scope.go:122] "RemoveContainer" containerID="c5995f589c0a666b8dc9dcd79862719add86f32442768980518654bb5c6ba77e" Oct 30 13:15:24.315708 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.301773 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz"] Oct 30 13:15:24.315708 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.305326 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3778dacd-5df6bbbb7b-h26tz"] Oct 30 13:15:24.353589 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:24.336666 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5995f589c0a666b8dc9dcd79862719add86f32442768980518654bb5c6ba77e\": container with ID starting with c5995f589c0a666b8dc9dcd79862719add86f32442768980518654bb5c6ba77e not found: ID does not exist" containerID="c5995f589c0a666b8dc9dcd79862719add86f32442768980518654bb5c6ba77e" Oct 30 13:15:24.353589 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.336718 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5995f589c0a666b8dc9dcd79862719add86f32442768980518654bb5c6ba77e"} err="failed to get container status \"c5995f589c0a666b8dc9dcd79862719add86f32442768980518654bb5c6ba77e\": rpc error: code = NotFound desc = could not find container \"c5995f589c0a666b8dc9dcd79862719add86f32442768980518654bb5c6ba77e\": container with ID starting with c5995f589c0a666b8dc9dcd79862719add86f32442768980518654bb5c6ba77e not found: ID does not exist" Oct 30 13:15:24.483639 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.481109 7010 scope.go:122] "RemoveContainer" containerID="4ad3c784013b08b56efaca0e198a74d3da828c2905bc544e999ca60a8bc364fb" Oct 30 13:15:24.483639 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.481251 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" Oct 30 13:15:24.593784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.588445 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0aace59-a54a-447a-ba91-3ab588b2c2fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0aace59-a54a-447a-ba91-3ab588b2c2fe-maas-api-service-ca\") pod \"a0aace59-a54a-447a-ba91-3ab588b2c2fe\" (UID: \"a0aace59-a54a-447a-ba91-3ab588b2c2fe\") " Oct 30 13:15:24.593784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.588552 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0aace59-a54a-447a-ba91-3ab588b2c2fe-kube-api-access-5hztm\" (UniqueName: \"kubernetes.io/projected/a0aace59-a54a-447a-ba91-3ab588b2c2fe-kube-api-access-5hztm\") pod \"a0aace59-a54a-447a-ba91-3ab588b2c2fe\" (UID: \"a0aace59-a54a-447a-ba91-3ab588b2c2fe\") " Oct 30 13:15:24.593784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.590237 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0aace59-a54a-447a-ba91-3ab588b2c2fe-maas-api-service-ca" pod "a0aace59-a54a-447a-ba91-3ab588b2c2fe" (UID: "a0aace59-a54a-447a-ba91-3ab588b2c2fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:24.639821 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.632921 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0aace59-a54a-447a-ba91-3ab588b2c2fe-kube-api-access-5hztm" pod "a0aace59-a54a-447a-ba91-3ab588b2c2fe" (UID: "a0aace59-a54a-447a-ba91-3ab588b2c2fe"). InnerVolumeSpecName "kube-api-access-5hztm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:24.692531 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.691803 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hztm\" (UniqueName: \"kubernetes.io/projected/a0aace59-a54a-447a-ba91-3ab588b2c2fe-kube-api-access-5hztm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:24.692531 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.691841 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0aace59-a54a-447a-ba91-3ab588b2c2fe-maas-api-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:24.792418 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.792382 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp"] Oct 30 13:15:24.798157 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:15:24.798130 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f248e80_d54a_44e8_b2b1_8ffd39b09e2f.slice/crio-2a6e7df538b973760eb1a96083441d75e81ad6a98ec95ee4d1c876cf4d073548 WatchSource:0}: Error finding container 2a6e7df538b973760eb1a96083441d75e81ad6a98ec95ee4d1c876cf4d073548: Status 404 returned error can't find the container with id 2a6e7df538b973760eb1a96083441d75e81ad6a98ec95ee4d1c876cf4d073548 Oct 30 13:15:24.799825 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.799802 7010 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:15:24.887403 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.887351 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" event={"ID":"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f","Type":"ContainerStarted","Data":"2a6e7df538b973760eb1a96083441d75e81ad6a98ec95ee4d1c876cf4d073548"} Oct 30 13:15:24.890247 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.890209 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" event={"ID":"9d86f05b-db3f-417f-b5ab-395ebcbb4a0e","Type":"ContainerStarted","Data":"701c447826197af749646e8d6c26c055a1b4365a28dbfe4fd69714c56c3b0301"} Oct 30 13:15:24.892933 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.892889 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" Oct 30 13:15:24.893132 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.892950 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6" event={"ID":"a0aace59-a54a-447a-ba91-3ab588b2c2fe","Type":"ContainerDied","Data":"383664d230b0a28a1d6b7ef26a4d29da7e0745dbe3528bd2f5b8eea6183cc5d8"} Oct 30 13:15:24.893132 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:24.893079 7010 scope.go:122] "RemoveContainer" containerID="64706c1906454d880171a078a347aaf750dc8bab675c101992dd42f36fd76ec9" Oct 30 13:15:24.904932 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:24.904848 7010 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice/crio-2a4854682bc850060336bd6497bf4f91fa6dce2a90552b13d418c0f0852fe925\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:15:25.008022 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:25.007991 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6"] Oct 30 13:15:25.011589 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:25.011565 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b29df4-29889435-mxkd6"] Oct 30 13:15:25.491954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:25.491927 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e792233-9490-4134-acce-5584ab43123a" path="/var/lib/kubelet/pods/3e792233-9490-4134-acce-5584ab43123a/volumes" Oct 30 13:15:25.492518 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:25.492498 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d4b02d9-cc6d-4783-8c2f-c9c39215e18c" path="/var/lib/kubelet/pods/5d4b02d9-cc6d-4783-8c2f-c9c39215e18c/volumes" Oct 30 13:15:25.492961 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:25.492946 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0aace59-a54a-447a-ba91-3ab588b2c2fe" path="/var/lib/kubelet/pods/a0aace59-a54a-447a-ba91-3ab588b2c2fe/volumes" Oct 30 13:15:25.903093 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:25.903050 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" event={"ID":"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f","Type":"ContainerStarted","Data":"5dad2fae5409780b751ebd4063cb5b5767c481aa4331b1cdebef7c816257622f"} Oct 30 13:15:25.903560 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:25.903116 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:15:25.905003 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:25.904977 7010 generic.go:356] "Generic (PLEG): container finished" podID="9ede3d71-cc65-4660-b836-6f05f27ea6cc" containerID="a3837a029b5bf18543ef5a6f57eb16f50d1977f16bb0e5974ef6e20af076b2b0" exitCode=60 Oct 30 13:15:25.905116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:25.905050 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" event={"ID":"9ede3d71-cc65-4660-b836-6f05f27ea6cc","Type":"ContainerDied","Data":"a3837a029b5bf18543ef5a6f57eb16f50d1977f16bb0e5974ef6e20af076b2b0"} Oct 30 13:15:25.905116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:25.905105 7010 scope.go:122] "RemoveContainer" containerID="4ad3c784013b08b56efaca0e198a74d3da828c2905bc544e999ca60a8bc364fb" Oct 30 13:15:25.905553 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:25.905523 7010 scope.go:122] "RemoveContainer" containerID="a3837a029b5bf18543ef5a6f57eb16f50d1977f16bb0e5974ef6e20af076b2b0" Oct 30 13:15:25.905917 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:25.905896 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889435-dlkxh_odh-ai-gateway-infra(9ede3d71-cc65-4660-b836-6f05f27ea6cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" podUID="9ede3d71-cc65-4660-b836-6f05f27ea6cc" Oct 30 13:15:25.907126 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:25.907106 7010 generic.go:356] "Generic (PLEG): container finished" podID="9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" containerID="701c447826197af749646e8d6c26c055a1b4365a28dbfe4fd69714c56c3b0301" exitCode=60 Oct 30 13:15:25.907223 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:25.907169 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" event={"ID":"9d86f05b-db3f-417f-b5ab-395ebcbb4a0e","Type":"ContainerDied","Data":"701c447826197af749646e8d6c26c055a1b4365a28dbfe4fd69714c56c3b0301"} Oct 30 13:15:25.907559 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:25.907546 7010 scope.go:122] "RemoveContainer" containerID="701c447826197af749646e8d6c26c055a1b4365a28dbfe4fd69714c56c3b0301" Oct 30 13:15:25.907861 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:25.907842 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp_odh-ai-gateway-infra(9d86f05b-db3f-417f-b5ab-395ebcbb4a0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" podUID="9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" Oct 30 13:15:25.922594 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:25.922576 7010 scope.go:122] "RemoveContainer" containerID="771238082f9b17f025280b04036f194da6c253099a8c9bdca5f94d04378c5aee" Oct 30 13:15:25.956358 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:25.956301 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" podStartSLOduration=19.592336468 podStartE2EDuration="19.956285275s" totalImagesPullingTime="363.948807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:25.955130852 +0000 UTC m=+1382.587318040" watchObservedRunningTime="2026-10-30 13:15:25.956285275 +0000 UTC m=+1382.588472437" Oct 30 13:15:25.977368 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:25.977331 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp"] Oct 30 13:15:27.022167 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.022121 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh"] Oct 30 13:15:27.119647 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.119613 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" Oct 30 13:15:27.232612 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.232587 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" Oct 30 13:15:27.240360 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.240329 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d86f05b-db3f-417f-b5ab-395ebcbb4a0e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d86f05b-db3f-417f-b5ab-395ebcbb4a0e-maas-api-service-ca\") pod \"9d86f05b-db3f-417f-b5ab-395ebcbb4a0e\" (UID: \"9d86f05b-db3f-417f-b5ab-395ebcbb4a0e\") " Oct 30 13:15:27.240514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.240483 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d86f05b-db3f-417f-b5ab-395ebcbb4a0e-kube-api-access-6db5q\" (UniqueName: \"kubernetes.io/projected/9d86f05b-db3f-417f-b5ab-395ebcbb4a0e-kube-api-access-6db5q\") pod \"9d86f05b-db3f-417f-b5ab-395ebcbb4a0e\" (UID: \"9d86f05b-db3f-417f-b5ab-395ebcbb4a0e\") " Oct 30 13:15:27.240785 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.240760 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d86f05b-db3f-417f-b5ab-395ebcbb4a0e-maas-api-service-ca" pod "9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" (UID: "9d86f05b-db3f-417f-b5ab-395ebcbb4a0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:27.241213 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.241194 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d86f05b-db3f-417f-b5ab-395ebcbb4a0e-maas-api-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:27.242524 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.242470 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d86f05b-db3f-417f-b5ab-395ebcbb4a0e-kube-api-access-6db5q" pod "9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" (UID: "9d86f05b-db3f-417f-b5ab-395ebcbb4a0e"). InnerVolumeSpecName "kube-api-access-6db5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:27.342785 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.342737 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ede3d71-cc65-4660-b836-6f05f27ea6cc-kube-api-access-j9pcq\" (UniqueName: \"kubernetes.io/projected/9ede3d71-cc65-4660-b836-6f05f27ea6cc-kube-api-access-j9pcq\") pod \"9ede3d71-cc65-4660-b836-6f05f27ea6cc\" (UID: \"9ede3d71-cc65-4660-b836-6f05f27ea6cc\") " Oct 30 13:15:27.342975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.342855 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ede3d71-cc65-4660-b836-6f05f27ea6cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ede3d71-cc65-4660-b836-6f05f27ea6cc-maas-api-service-ca\") pod \"9ede3d71-cc65-4660-b836-6f05f27ea6cc\" (UID: \"9ede3d71-cc65-4660-b836-6f05f27ea6cc\") " Oct 30 13:15:27.343575 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.343541 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ede3d71-cc65-4660-b836-6f05f27ea6cc-maas-api-service-ca" pod "9ede3d71-cc65-4660-b836-6f05f27ea6cc" (UID: "9ede3d71-cc65-4660-b836-6f05f27ea6cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:27.343753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.343732 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6db5q\" (UniqueName: \"kubernetes.io/projected/9d86f05b-db3f-417f-b5ab-395ebcbb4a0e-kube-api-access-6db5q\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:27.343803 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.343759 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ede3d71-cc65-4660-b836-6f05f27ea6cc-maas-api-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:27.345841 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.345818 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ede3d71-cc65-4660-b836-6f05f27ea6cc-kube-api-access-j9pcq" pod "9ede3d71-cc65-4660-b836-6f05f27ea6cc" (UID: "9ede3d71-cc65-4660-b836-6f05f27ea6cc"). InnerVolumeSpecName "kube-api-access-j9pcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:27.445014 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.444968 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9pcq\" (UniqueName: \"kubernetes.io/projected/9ede3d71-cc65-4660-b836-6f05f27ea6cc-kube-api-access-j9pcq\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:27.926080 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.926047 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" Oct 30 13:15:27.926275 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.926044 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh" event={"ID":"9ede3d71-cc65-4660-b836-6f05f27ea6cc","Type":"ContainerDied","Data":"d22c50035991cd3ca72623ca149d4480e8d10102a8ee56f2801025104110a23f"} Oct 30 13:15:27.926275 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.926169 7010 scope.go:122] "RemoveContainer" containerID="a3837a029b5bf18543ef5a6f57eb16f50d1977f16bb0e5974ef6e20af076b2b0" Oct 30 13:15:27.927520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.927469 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" Oct 30 13:15:27.927618 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.927522 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp" event={"ID":"9d86f05b-db3f-417f-b5ab-395ebcbb4a0e","Type":"ContainerDied","Data":"f00057a033dcbec26b5c936bbcff2cb87abbd3fa34c95898b6a84acc7bce7a46"} Oct 30 13:15:27.936690 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.936664 7010 scope.go:122] "RemoveContainer" containerID="701c447826197af749646e8d6c26c055a1b4365a28dbfe4fd69714c56c3b0301" Oct 30 13:15:27.981470 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.981434 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp"] Oct 30 13:15:27.984932 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:27.984905 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fafb3f-29889435-lzjzp"] Oct 30 13:15:28.024309 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:28.024276 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh"] Oct 30 13:15:28.032530 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:28.032499 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889435-dlkxh"] Oct 30 13:15:29.401637 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:29.401596 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:15:29.489994 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:29.489959 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" path="/var/lib/kubelet/pods/9d86f05b-db3f-417f-b5ab-395ebcbb4a0e/volumes" Oct 30 13:15:29.490501 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:29.490472 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ede3d71-cc65-4660-b836-6f05f27ea6cc" path="/var/lib/kubelet/pods/9ede3d71-cc65-4660-b836-6f05f27ea6cc/volumes" Oct 30 13:15:31.077713 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:31.077674 7010 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice/crio-2a4854682bc850060336bd6497bf4f91fa6dce2a90552b13d418c0f0852fe925\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:15:31.923617 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:31.923588 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:15:32.223449 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:32.223362 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk"] Oct 30 13:15:32.223923 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:32.223633 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" podUID="428e36fe-a829-4bee-bcd5-5150d5358411" containerName="istio-proxy" containerID="cri-o://e8ec7090dd84e413dde6d4f90fda910a406f0c2039b9191bc45251a063290999" gracePeriod=30 Oct 30 13:15:37.518471 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.518441 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:15:37.631402 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.631363 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-workload-certs\") pod \"428e36fe-a829-4bee-bcd5-5150d5358411\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " Oct 30 13:15:37.631614 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.631410 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428e36fe-a829-4bee-bcd5-5150d5358411-kube-api-access-xbnmj\" (UniqueName: \"kubernetes.io/projected/428e36fe-a829-4bee-bcd5-5150d5358411-kube-api-access-xbnmj\") pod \"428e36fe-a829-4bee-bcd5-5150d5358411\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " Oct 30 13:15:37.631614 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.631462 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-istio-envoy\") pod \"428e36fe-a829-4bee-bcd5-5150d5358411\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " Oct 30 13:15:37.631614 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.631515 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/428e36fe-a829-4bee-bcd5-5150d5358411-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/428e36fe-a829-4bee-bcd5-5150d5358411-istio-podinfo\") pod \"428e36fe-a829-4bee-bcd5-5150d5358411\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " Oct 30 13:15:37.631614 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.631574 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-workload-socket\") pod \"428e36fe-a829-4bee-bcd5-5150d5358411\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " Oct 30 13:15:37.631858 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.631624 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428e36fe-a829-4bee-bcd5-5150d5358411-istio-token\" (UniqueName: \"kubernetes.io/projected/428e36fe-a829-4bee-bcd5-5150d5358411-istio-token\") pod \"428e36fe-a829-4bee-bcd5-5150d5358411\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " Oct 30 13:15:37.631858 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.631666 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428e36fe-a829-4bee-bcd5-5150d5358411-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/428e36fe-a829-4bee-bcd5-5150d5358411-istiod-ca-cert\") pod \"428e36fe-a829-4bee-bcd5-5150d5358411\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " Oct 30 13:15:37.631858 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.631722 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-credential-socket\") pod \"428e36fe-a829-4bee-bcd5-5150d5358411\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " Oct 30 13:15:37.632019 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.631944 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-istio-data\") pod \"428e36fe-a829-4bee-bcd5-5150d5358411\" (UID: \"428e36fe-a829-4bee-bcd5-5150d5358411\") " Oct 30 13:15:37.632483 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.631719 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-workload-certs" pod "428e36fe-a829-4bee-bcd5-5150d5358411" (UID: "428e36fe-a829-4bee-bcd5-5150d5358411"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:15:37.632483 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.632091 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428e36fe-a829-4bee-bcd5-5150d5358411-istiod-ca-cert" pod "428e36fe-a829-4bee-bcd5-5150d5358411" (UID: "428e36fe-a829-4bee-bcd5-5150d5358411"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:15:37.632483 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.632315 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-credential-socket" pod "428e36fe-a829-4bee-bcd5-5150d5358411" (UID: "428e36fe-a829-4bee-bcd5-5150d5358411"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:15:37.632483 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.632432 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-istio-data" pod "428e36fe-a829-4bee-bcd5-5150d5358411" (UID: "428e36fe-a829-4bee-bcd5-5150d5358411"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:15:37.632483 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.632449 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-workload-socket" pod "428e36fe-a829-4bee-bcd5-5150d5358411" (UID: "428e36fe-a829-4bee-bcd5-5150d5358411"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:15:37.632921 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.632898 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:37.632997 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.632924 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:37.632997 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.632939 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/428e36fe-a829-4bee-bcd5-5150d5358411-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:37.632997 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.632953 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:37.632997 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.632968 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:37.634816 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.634739 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428e36fe-a829-4bee-bcd5-5150d5358411-istio-token" pod "428e36fe-a829-4bee-bcd5-5150d5358411" (UID: "428e36fe-a829-4bee-bcd5-5150d5358411"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:37.635288 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.635268 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-istio-envoy" pod "428e36fe-a829-4bee-bcd5-5150d5358411" (UID: "428e36fe-a829-4bee-bcd5-5150d5358411"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:15:37.635470 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.635427 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/428e36fe-a829-4bee-bcd5-5150d5358411-istio-podinfo" pod "428e36fe-a829-4bee-bcd5-5150d5358411" (UID: "428e36fe-a829-4bee-bcd5-5150d5358411"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:15:37.635758 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.635730 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428e36fe-a829-4bee-bcd5-5150d5358411-kube-api-access-xbnmj" pod "428e36fe-a829-4bee-bcd5-5150d5358411" (UID: "428e36fe-a829-4bee-bcd5-5150d5358411"). InnerVolumeSpecName "kube-api-access-xbnmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:15:37.734525 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.734477 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbnmj\" (UniqueName: \"kubernetes.io/projected/428e36fe-a829-4bee-bcd5-5150d5358411-kube-api-access-xbnmj\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:37.734525 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.734519 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/428e36fe-a829-4bee-bcd5-5150d5358411-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:37.734525 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.734529 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/428e36fe-a829-4bee-bcd5-5150d5358411-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:37.734757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:37.734538 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/428e36fe-a829-4bee-bcd5-5150d5358411-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:15:38.021590 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:38.021553 7010 generic.go:356] "Generic (PLEG): container finished" podID="428e36fe-a829-4bee-bcd5-5150d5358411" containerID="e8ec7090dd84e413dde6d4f90fda910a406f0c2039b9191bc45251a063290999" exitCode=0 Oct 30 13:15:38.021790 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:38.021605 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" Oct 30 13:15:38.021790 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:38.021635 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" event={"ID":"428e36fe-a829-4bee-bcd5-5150d5358411","Type":"ContainerDied","Data":"e8ec7090dd84e413dde6d4f90fda910a406f0c2039b9191bc45251a063290999"} Oct 30 13:15:38.021790 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:38.021697 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk" event={"ID":"428e36fe-a829-4bee-bcd5-5150d5358411","Type":"ContainerDied","Data":"729f86345c78928af8cf83fbba445f5e51eb194d9e5e9fce7742e301b13b7f08"} Oct 30 13:15:38.021790 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:38.021727 7010 scope.go:122] "RemoveContainer" containerID="e8ec7090dd84e413dde6d4f90fda910a406f0c2039b9191bc45251a063290999" Oct 30 13:15:38.033131 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:38.033115 7010 scope.go:122] "RemoveContainer" containerID="7360cc0b840e330e07e05af9d2af3e4e62d4efc51841d2b2034a81ca71be709f" Oct 30 13:15:38.042247 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:38.042226 7010 scope.go:122] "RemoveContainer" containerID="e8ec7090dd84e413dde6d4f90fda910a406f0c2039b9191bc45251a063290999" Oct 30 13:15:38.042562 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:38.042542 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ec7090dd84e413dde6d4f90fda910a406f0c2039b9191bc45251a063290999\": container with ID starting with e8ec7090dd84e413dde6d4f90fda910a406f0c2039b9191bc45251a063290999 not found: ID does not exist" containerID="e8ec7090dd84e413dde6d4f90fda910a406f0c2039b9191bc45251a063290999" Oct 30 13:15:38.042611 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:38.042583 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ec7090dd84e413dde6d4f90fda910a406f0c2039b9191bc45251a063290999"} err="failed to get container status \"e8ec7090dd84e413dde6d4f90fda910a406f0c2039b9191bc45251a063290999\": rpc error: code = NotFound desc = could not find container \"e8ec7090dd84e413dde6d4f90fda910a406f0c2039b9191bc45251a063290999\": container with ID starting with e8ec7090dd84e413dde6d4f90fda910a406f0c2039b9191bc45251a063290999 not found: ID does not exist" Oct 30 13:15:38.042611 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:38.042601 7010 scope.go:122] "RemoveContainer" containerID="7360cc0b840e330e07e05af9d2af3e4e62d4efc51841d2b2034a81ca71be709f" Oct 30 13:15:38.042837 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:38.042820 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7360cc0b840e330e07e05af9d2af3e4e62d4efc51841d2b2034a81ca71be709f\": container with ID starting with 7360cc0b840e330e07e05af9d2af3e4e62d4efc51841d2b2034a81ca71be709f not found: ID does not exist" containerID="7360cc0b840e330e07e05af9d2af3e4e62d4efc51841d2b2034a81ca71be709f" Oct 30 13:15:38.042882 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:38.042843 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7360cc0b840e330e07e05af9d2af3e4e62d4efc51841d2b2034a81ca71be709f"} err="failed to get container status \"7360cc0b840e330e07e05af9d2af3e4e62d4efc51841d2b2034a81ca71be709f\": rpc error: code = NotFound desc = could not find container \"7360cc0b840e330e07e05af9d2af3e4e62d4efc51841d2b2034a81ca71be709f\": container with ID starting with 7360cc0b840e330e07e05af9d2af3e4e62d4efc51841d2b2034a81ca71be709f not found: ID does not exist" Oct 30 13:15:38.090011 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:38.089985 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk"] Oct 30 13:15:38.102176 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:38.102149 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3778dacd-openshift-default-55b77595bf-drjgk"] Oct 30 13:15:39.492698 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:39.492651 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428e36fe-a829-4bee-bcd5-5150d5358411" path="/var/lib/kubelet/pods/428e36fe-a829-4bee-bcd5-5150d5358411/volumes" Oct 30 13:15:39.908026 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:39.907987 7010 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice/crio-2a4854682bc850060336bd6497bf4f91fa6dce2a90552b13d418c0f0852fe925\": RecentStats: unable to find data in memory cache]" Oct 30 13:15:41.096398 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:41.096352 7010 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice/crio-2a4854682bc850060336bd6497bf4f91fa6dce2a90552b13d418c0f0852fe925\": RecentStats: unable to find data in memory cache]" Oct 30 13:15:51.190923 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:51.190880 7010 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice/crio-2a4854682bc850060336bd6497bf4f91fa6dce2a90552b13d418c0f0852fe925\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:15:51.982623 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:51.982590 7010 patch_prober.go:29] interesting pod/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:15:51.982890 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:51.982651 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" podUID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:15:54.395110 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:54.395071 7010 generic.go:356] "Generic (PLEG): container finished" podID="3575f061-f8d0-4915-b4e8-c88c882407e5" containerID="d4629e7d65510a170b67d2a8e9336306456a592326d8dd6387f9ee30a3a4a799" exitCode=0 Oct 30 13:15:54.395622 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:54.395151 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" event={"ID":"3575f061-f8d0-4915-b4e8-c88c882407e5","Type":"ContainerDied","Data":"d4629e7d65510a170b67d2a8e9336306456a592326d8dd6387f9ee30a3a4a799"} Oct 30 13:15:54.395744 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:54.395722 7010 scope.go:122] "RemoveContainer" containerID="d4629e7d65510a170b67d2a8e9336306456a592326d8dd6387f9ee30a3a4a799" Oct 30 13:15:54.397162 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:54.397140 7010 generic.go:356] "Generic (PLEG): container finished" podID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" containerID="bb1e7b1a7e514b51b265d1829973e1c40f07091431f97f892bd5d5df44849fe0" exitCode=0 Oct 30 13:15:54.397272 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:54.397192 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" event={"ID":"c0a3ed5e-a26f-4514-921d-24b409b1dc27","Type":"ContainerDied","Data":"bb1e7b1a7e514b51b265d1829973e1c40f07091431f97f892bd5d5df44849fe0"} Oct 30 13:15:54.397658 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:54.397639 7010 scope.go:122] "RemoveContainer" containerID="bb1e7b1a7e514b51b265d1829973e1c40f07091431f97f892bd5d5df44849fe0" Oct 30 13:15:54.397779 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:54.397752 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:15:54.397847 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:54.397820 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:15:54.397898 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:54.397850 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:15:54.399207 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:54.399177 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:15:54.399312 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:54.399253 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:15:54.399312 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:54.399302 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:15:54.825016 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:54.824909 7010 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice/crio-2a4854682bc850060336bd6497bf4f91fa6dce2a90552b13d418c0f0852fe925\": RecentStats: unable to find data in memory cache]" Oct 30 13:15:55.429510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.424692 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" event={"ID":"3575f061-f8d0-4915-b4e8-c88c882407e5","Type":"ContainerStarted","Data":"b1a0ce1ffc50f11a1e2d7bc3aeb9352a1d93208e3f45e5e945693f044ade7dde"} Oct 30 13:15:55.447522 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.446730 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" event={"ID":"c0a3ed5e-a26f-4514-921d-24b409b1dc27","Type":"ContainerStarted","Data":"35aafcc6a759bb96be6d83de18da24367b5473bf0e623a0f02cfd7820d9bb0f4"} Oct 30 13:15:55.751627 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.751543 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:15:55.754661 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.754611 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49"] Oct 30 13:15:55.755373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.755343 7010 patch_prober.go:29] interesting pod/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:15:55.755476 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.755412 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" podUID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 13:15:55.756044 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.756022 7010 cpu_manager.go:414] "containerMap: removing container" podUID="428e36fe-a829-4bee-bcd5-5150d5358411" containerName="istio-proxy" Oct 30 13:15:55.756158 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.756144 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428e36fe-a829-4bee-bcd5-5150d5358411" containerName="istio-proxy" Oct 30 13:15:55.756288 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.756275 7010 cpu_manager.go:414] "containerMap: removing container" podUID="9ede3d71-cc65-4660-b836-6f05f27ea6cc" containerName="cleanup" Oct 30 13:15:55.756374 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.756363 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ede3d71-cc65-4660-b836-6f05f27ea6cc" containerName="cleanup" Oct 30 13:15:55.756462 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.756452 7010 cpu_manager.go:414] "containerMap: removing container" podUID="a0aace59-a54a-447a-ba91-3ab588b2c2fe" containerName="cleanup" Oct 30 13:15:55.756561 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.756549 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0aace59-a54a-447a-ba91-3ab588b2c2fe" containerName="cleanup" Oct 30 13:15:55.756647 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.756636 7010 cpu_manager.go:414] "containerMap: removing container" podUID="9ede3d71-cc65-4660-b836-6f05f27ea6cc" containerName="cleanup" Oct 30 13:15:55.756723 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.756712 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ede3d71-cc65-4660-b836-6f05f27ea6cc" containerName="cleanup" Oct 30 13:15:55.756833 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.756823 7010 cpu_manager.go:414] "containerMap: removing container" podUID="9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" containerName="cleanup" Oct 30 13:15:55.756902 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.756893 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" containerName="cleanup" Oct 30 13:15:55.756983 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.756974 7010 cpu_manager.go:414] "containerMap: removing container" podUID="9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" containerName="cleanup" Oct 30 13:15:55.757054 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.757045 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" containerName="cleanup" Oct 30 13:15:55.757128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.757119 7010 cpu_manager.go:414] "containerMap: removing container" podUID="a0aace59-a54a-447a-ba91-3ab588b2c2fe" containerName="cleanup" Oct 30 13:15:55.757215 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.757204 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0aace59-a54a-447a-ba91-3ab588b2c2fe" containerName="cleanup" Oct 30 13:15:55.757297 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.757287 7010 cpu_manager.go:414] "containerMap: removing container" podUID="9ede3d71-cc65-4660-b836-6f05f27ea6cc" containerName="cleanup" Oct 30 13:15:55.757364 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.757355 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ede3d71-cc65-4660-b836-6f05f27ea6cc" containerName="cleanup" Oct 30 13:15:55.757457 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.757439 7010 cpu_manager.go:414] "containerMap: removing container" podUID="9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" containerName="cleanup" Oct 30 13:15:55.757562 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.757551 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" containerName="cleanup" Oct 30 13:15:55.757859 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.757841 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="428e36fe-a829-4bee-bcd5-5150d5358411" containerName="istio-proxy" Oct 30 13:15:55.757961 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.757952 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" containerName="cleanup" Oct 30 13:15:55.758066 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.758055 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0aace59-a54a-447a-ba91-3ab588b2c2fe" containerName="cleanup" Oct 30 13:15:55.758159 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.758149 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ede3d71-cc65-4660-b836-6f05f27ea6cc" containerName="cleanup" Oct 30 13:15:55.758242 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.758233 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ede3d71-cc65-4660-b836-6f05f27ea6cc" containerName="cleanup" Oct 30 13:15:55.758323 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.758313 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" containerName="cleanup" Oct 30 13:15:55.758402 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.758393 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0aace59-a54a-447a-ba91-3ab588b2c2fe" containerName="cleanup" Oct 30 13:15:55.769603 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.769584 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49"] Oct 30 13:15:55.769767 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.769747 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:55.801610 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.792117 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-cda0d9-openshift-default-dockercfg-jlswf\"" Oct 30 13:15:55.802112 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.802087 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:15:55.848514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.804851 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:15:55.848514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.839200 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" podUID="3575f061-f8d0-4915-b4e8-c88c882407e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 13:15:55.894511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.889262 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj86n\" (UniqueName: \"kubernetes.io/projected/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-kube-api-access-vj86n\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:55.894511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.889433 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-token\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:55.894511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.889475 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-workload-certs\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:55.894511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.889617 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istiod-ca-cert\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:55.894511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.889737 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-podinfo\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:55.894511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.889775 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-data\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:55.980628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.980593 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-credential-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:55.980944 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.980924 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-workload-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:55.992701 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:55.992667 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-envoy\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.099392 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.097664 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-envoy\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.099392 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.097771 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj86n\" (UniqueName: \"kubernetes.io/projected/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-kube-api-access-vj86n\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.099392 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.097845 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-token\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.099392 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.097878 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-workload-certs\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.099392 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.097917 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istiod-ca-cert\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.099392 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.097953 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-podinfo\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.099392 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.097990 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-data\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.099392 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.098201 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-credential-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.099392 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.098271 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-workload-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.099392 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.098825 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-workload-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.107507 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.102216 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istiod-ca-cert\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.120534 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.115806 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-data\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.120534 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.116191 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-workload-certs\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.120534 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.116439 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-credential-socket\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.139695 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.132074 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-envoy\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.146513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.142549 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-podinfo\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.191523 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.182554 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-token\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.198524 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.192242 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj86n\" (UniqueName: \"kubernetes.io/projected/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-kube-api-access-vj86n\") pod \"e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.312378 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.310027 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:56.712768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.712653 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:15:56.745518 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.737900 7010 patch_prober.go:29] interesting pod/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:15:56.745518 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.737967 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" podUID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 13:15:56.800514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.798893 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:15:56.800514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:56.798954 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" podUID="3575f061-f8d0-4915-b4e8-c88c882407e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 13:15:57.117571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:57.114107 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49"] Oct 30 13:15:57.133157 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:57.133121 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:15:57.133376 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:57.133353 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:15:57.133561 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:57.133538 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:15:57.537868 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:57.537826 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" event={"ID":"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2","Type":"ContainerStarted","Data":"9f2ac25441e81cecfcb7c148a0eadc29f0a1bd6e747908624cd275d249d52ec0"} Oct 30 13:15:57.738164 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:57.738059 7010 patch_prober.go:29] interesting pod/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:15:57.738164 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:57.738142 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" podUID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 13:15:57.810679 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:57.803929 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:15:57.810679 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:57.804011 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" podUID="3575f061-f8d0-4915-b4e8-c88c882407e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 13:15:58.550836 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:58.550791 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" event={"ID":"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2","Type":"ContainerStarted","Data":"d7ae7821917dc2438911ded9942372ca66f915b8ef34434bf5aafac3c9d0a874"} Oct 30 13:15:58.644139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:58.644056 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" podStartSLOduration=3.644036092 podStartE2EDuration="3.644036092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:58.641092487 +0000 UTC m=+1415.273279649" watchObservedRunningTime="2026-10-30 13:15:58.644036092 +0000 UTC m=+1415.276223255" Oct 30 13:15:58.753984 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:58.753947 7010 patch_prober.go:29] interesting pod/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:15:58.754659 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:58.754626 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" podUID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 13:15:58.796220 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:58.796180 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:15:58.796553 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:58.796517 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" podUID="3575f061-f8d0-4915-b4e8-c88c882407e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 13:15:59.317121 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:59.317084 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:59.330517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:59.326764 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:59.468689 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:15:59.467143 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:15:59.559861 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:59.559824 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:59.562693 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:59.562670 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:15:59.773514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:59.771620 7010 patch_prober.go:29] interesting pod/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:15:59.773514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:59.771680 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" podUID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 13:15:59.847559 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:59.842942 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:15:59.847559 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:15:59.843020 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" podUID="3575f061-f8d0-4915-b4e8-c88c882407e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 13:16:00.737125 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:00.737083 7010 patch_prober.go:29] interesting pod/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:16:00.737362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:00.737141 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" podUID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 13:16:00.804106 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:00.804056 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:16:00.804797 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:00.804764 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" podUID="3575f061-f8d0-4915-b4e8-c88c882407e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 13:16:01.875514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:01.813891 7010 patch_prober.go:29] interesting pod/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 13:16:01.875514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:01.813988 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" podUID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 13:16:01.875514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:01.815083 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 13:16:01.875514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:01.815135 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" podUID="3575f061-f8d0-4915-b4e8-c88c882407e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 13:16:02.648313 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:02.647482 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:16:02.746515 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:02.744044 7010 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice/crio-2a4854682bc850060336bd6497bf4f91fa6dce2a90552b13d418c0f0852fe925\": RecentStats: unable to find data in memory cache]" Oct 30 13:16:03.172510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:03.168448 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:16:03.173659 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:03.173397 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:16:03.331777 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:03.331741 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:16:03.332218 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:03.332163 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:16:03.336553 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:03.336528 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:16:09.888465 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:09.888417 7010 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice/crio-2a4854682bc850060336bd6497bf4f91fa6dce2a90552b13d418c0f0852fe925\": RecentStats: unable to find data in memory cache]" Oct 30 13:16:10.047501 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:10.047450 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29"] Oct 30 13:16:10.047738 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:10.047695 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" podUID="d5dacfc7-1c21-4ac5-b93a-1f0763af424f" containerName="istio-proxy" containerID="cri-o://b6ad17091fa82f57b2459a93cc957fd3579b189e7e811bb685ee4a4496d026e0" gracePeriod=30 Oct 30 13:16:10.060967 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:10.060932 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx"] Oct 30 13:16:10.061201 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:10.061175 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" podUID="ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" containerName="istio-proxy" containerID="cri-o://5081caa1009928136a004e10a2777239f3c4ffc1f5d8e606b1a770d947f664c0" gracePeriod=30 Oct 30 13:16:12.793360 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:12.793322 7010 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice/crio-2a4854682bc850060336bd6497bf4f91fa6dce2a90552b13d418c0f0852fe925\": RecentStats: unable to find data in memory cache]" Oct 30 13:16:15.361117 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.361083 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:16:15.413291 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.413259 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:16:15.467889 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.467855 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-credential-socket\") pod \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " Oct 30 13:16:15.468072 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.467976 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-podinfo\") pod \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " Oct 30 13:16:15.468072 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.468008 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-token\" (UniqueName: \"kubernetes.io/projected/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-token\") pod \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " Oct 30 13:16:15.468072 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.468038 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-credential-socket\") pod \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " Oct 30 13:16:15.468264 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.468107 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-data\") pod \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " Oct 30 13:16:15.468264 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.468163 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-envoy\") pod \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " Oct 30 13:16:15.468264 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.468196 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-credential-socket" pod "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" (UID: "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:15.468264 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.468238 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-workload-certs\") pod \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " Oct 30 13:16:15.468480 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.468283 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istiod-ca-cert\") pod \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " Oct 30 13:16:15.468480 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.468334 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-podinfo\") pod \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " Oct 30 13:16:15.468480 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.468375 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-credential-socket" pod "d5dacfc7-1c21-4ac5-b93a-1f0763af424f" (UID: "d5dacfc7-1c21-4ac5-b93a-1f0763af424f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:15.468480 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.468432 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-workload-certs\") pod \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " Oct 30 13:16:15.468480 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.468464 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-kube-api-access-khlpc\" (UniqueName: \"kubernetes.io/projected/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-kube-api-access-khlpc\") pod \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " Oct 30 13:16:15.468480 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.468466 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-data" pod "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" (UID: "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:15.468864 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.468649 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-workload-certs" pod "d5dacfc7-1c21-4ac5-b93a-1f0763af424f" (UID: "d5dacfc7-1c21-4ac5-b93a-1f0763af424f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:15.468864 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.468679 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-workload-certs" pod "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" (UID: "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:15.469426 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.469400 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-envoy\") pod \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " Oct 30 13:16:15.469612 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.469534 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-workload-socket\") pod \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " Oct 30 13:16:15.469612 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.469584 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istiod-ca-cert\") pod \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " Oct 30 13:16:15.469743 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.469651 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-workload-socket\") pod \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\" (UID: \"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db\") " Oct 30 13:16:15.469743 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.469693 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-token\" (UniqueName: \"kubernetes.io/projected/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-token\") pod \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " Oct 30 13:16:15.469743 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.469735 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-data\") pod \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " Oct 30 13:16:15.469899 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.469820 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-kube-api-access-s2gl5\" (UniqueName: \"kubernetes.io/projected/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-kube-api-access-s2gl5\") pod \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\" (UID: \"d5dacfc7-1c21-4ac5-b93a-1f0763af424f\") " Oct 30 13:16:15.470903 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.470872 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istiod-ca-cert" pod "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" (UID: "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:15.470989 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.470947 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-podinfo" pod "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" (UID: "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:15.471463 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.471435 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-data" pod "d5dacfc7-1c21-4ac5-b93a-1f0763af424f" (UID: "d5dacfc7-1c21-4ac5-b93a-1f0763af424f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:15.471606 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.471467 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-workload-socket" pod "d5dacfc7-1c21-4ac5-b93a-1f0763af424f" (UID: "d5dacfc7-1c21-4ac5-b93a-1f0763af424f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:15.471746 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.471722 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-workload-socket" pod "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" (UID: "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:15.471882 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.471857 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:15.471950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.471888 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:15.471950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.471903 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:15.471950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.471916 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:15.471950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.471930 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:15.471950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.471945 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:15.472189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.471960 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:15.472189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.471975 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:15.472189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.471987 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:15.472189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.472150 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istiod-ca-cert" pod "d5dacfc7-1c21-4ac5-b93a-1f0763af424f" (UID: "d5dacfc7-1c21-4ac5-b93a-1f0763af424f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:15.472551 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.472526 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-kube-api-access-khlpc" pod "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" (UID: "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db"). InnerVolumeSpecName "kube-api-access-khlpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:15.472675 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.472621 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-podinfo" pod "d5dacfc7-1c21-4ac5-b93a-1f0763af424f" (UID: "d5dacfc7-1c21-4ac5-b93a-1f0763af424f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:15.473390 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.473344 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-envoy" pod "d5dacfc7-1c21-4ac5-b93a-1f0763af424f" (UID: "d5dacfc7-1c21-4ac5-b93a-1f0763af424f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:15.473390 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.473350 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-envoy" pod "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" (UID: "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:15.474415 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.474254 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-token" pod "d5dacfc7-1c21-4ac5-b93a-1f0763af424f" (UID: "d5dacfc7-1c21-4ac5-b93a-1f0763af424f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:15.480967 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.480602 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-token" pod "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" (UID: "ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:15.481091 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.481056 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-kube-api-access-s2gl5" pod "d5dacfc7-1c21-4ac5-b93a-1f0763af424f" (UID: "d5dacfc7-1c21-4ac5-b93a-1f0763af424f"). InnerVolumeSpecName "kube-api-access-s2gl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:15.573721 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.573640 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:15.573721 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.573670 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:15.573721 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.573679 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:15.573721 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.573691 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khlpc\" (UniqueName: \"kubernetes.io/projected/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-kube-api-access-khlpc\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:15.573721 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.573707 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:15.573721 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.573716 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:15.573721 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.573723 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:15.574062 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.573732 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2gl5\" (UniqueName: \"kubernetes.io/projected/d5dacfc7-1c21-4ac5-b93a-1f0763af424f-kube-api-access-s2gl5\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:15.574062 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:15.573741 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:16.032518 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.032469 7010 generic.go:356] "Generic (PLEG): container finished" podID="ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" containerID="5081caa1009928136a004e10a2777239f3c4ffc1f5d8e606b1a770d947f664c0" exitCode=0 Oct 30 13:16:16.032698 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.032538 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" Oct 30 13:16:16.032698 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.032540 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" event={"ID":"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db","Type":"ContainerDied","Data":"5081caa1009928136a004e10a2777239f3c4ffc1f5d8e606b1a770d947f664c0"} Oct 30 13:16:16.032698 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.032655 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx" event={"ID":"ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db","Type":"ContainerDied","Data":"41f15b917aa37ee1243637ef80f4dfeb701bb7febb2b8277c8f8d83f5f64e384"} Oct 30 13:16:16.032698 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.032686 7010 scope.go:122] "RemoveContainer" containerID="5081caa1009928136a004e10a2777239f3c4ffc1f5d8e606b1a770d947f664c0" Oct 30 13:16:16.034219 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.034193 7010 generic.go:356] "Generic (PLEG): container finished" podID="d5dacfc7-1c21-4ac5-b93a-1f0763af424f" containerID="b6ad17091fa82f57b2459a93cc957fd3579b189e7e811bb685ee4a4496d026e0" exitCode=0 Oct 30 13:16:16.034373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.034234 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" Oct 30 13:16:16.034373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.034254 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" event={"ID":"d5dacfc7-1c21-4ac5-b93a-1f0763af424f","Type":"ContainerDied","Data":"b6ad17091fa82f57b2459a93cc957fd3579b189e7e811bb685ee4a4496d026e0"} Oct 30 13:16:16.034373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.034292 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29" event={"ID":"d5dacfc7-1c21-4ac5-b93a-1f0763af424f","Type":"ContainerDied","Data":"76e9682e213da1bd7f0eae0189b593674430b78d6d40be5e75b310bfabad722e"} Oct 30 13:16:16.043649 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.043632 7010 scope.go:122] "RemoveContainer" containerID="5081caa1009928136a004e10a2777239f3c4ffc1f5d8e606b1a770d947f664c0" Oct 30 13:16:16.043924 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:16.043905 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5081caa1009928136a004e10a2777239f3c4ffc1f5d8e606b1a770d947f664c0\": container with ID starting with 5081caa1009928136a004e10a2777239f3c4ffc1f5d8e606b1a770d947f664c0 not found: ID does not exist" containerID="5081caa1009928136a004e10a2777239f3c4ffc1f5d8e606b1a770d947f664c0" Oct 30 13:16:16.043980 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.043933 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5081caa1009928136a004e10a2777239f3c4ffc1f5d8e606b1a770d947f664c0"} err="failed to get container status \"5081caa1009928136a004e10a2777239f3c4ffc1f5d8e606b1a770d947f664c0\": rpc error: code = NotFound desc = could not find container \"5081caa1009928136a004e10a2777239f3c4ffc1f5d8e606b1a770d947f664c0\": container with ID starting with 5081caa1009928136a004e10a2777239f3c4ffc1f5d8e606b1a770d947f664c0 not found: ID does not exist" Oct 30 13:16:16.043980 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.043956 7010 scope.go:122] "RemoveContainer" containerID="b6ad17091fa82f57b2459a93cc957fd3579b189e7e811bb685ee4a4496d026e0" Oct 30 13:16:16.053377 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.053354 7010 scope.go:122] "RemoveContainer" containerID="b6ad17091fa82f57b2459a93cc957fd3579b189e7e811bb685ee4a4496d026e0" Oct 30 13:16:16.053685 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:16.053665 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ad17091fa82f57b2459a93cc957fd3579b189e7e811bb685ee4a4496d026e0\": container with ID starting with b6ad17091fa82f57b2459a93cc957fd3579b189e7e811bb685ee4a4496d026e0 not found: ID does not exist" containerID="b6ad17091fa82f57b2459a93cc957fd3579b189e7e811bb685ee4a4496d026e0" Oct 30 13:16:16.053739 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.053695 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ad17091fa82f57b2459a93cc957fd3579b189e7e811bb685ee4a4496d026e0"} err="failed to get container status \"b6ad17091fa82f57b2459a93cc957fd3579b189e7e811bb685ee4a4496d026e0\": rpc error: code = NotFound desc = could not find container \"b6ad17091fa82f57b2459a93cc957fd3579b189e7e811bb685ee4a4496d026e0\": container with ID starting with b6ad17091fa82f57b2459a93cc957fd3579b189e7e811bb685ee4a4496d026e0 not found: ID does not exist" Oct 30 13:16:16.092757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.092720 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29"] Oct 30 13:16:16.100144 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:16.100112 7010 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice/crio-2a4854682bc850060336bd6497bf4f91fa6dce2a90552b13d418c0f0852fe925\": RecentStats: unable to find data in memory cache]" Oct 30 13:16:16.100144 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:16.100125 7010 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice/crio-2a4854682bc850060336bd6497bf4f91fa6dce2a90552b13d418c0f0852fe925\": RecentStats: unable to find data in memory cache]" Oct 30 13:16:16.103767 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.103742 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-xsj29"] Oct 30 13:16:16.130074 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.130036 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx"] Oct 30 13:16:16.134406 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:16.134376 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-b27jx"] Oct 30 13:16:17.490988 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:17.490957 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" path="/var/lib/kubelet/pods/ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db/volumes" Oct 30 13:16:17.491587 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:17.491571 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5dacfc7-1c21-4ac5-b93a-1f0763af424f" path="/var/lib/kubelet/pods/d5dacfc7-1c21-4ac5-b93a-1f0763af424f/volumes" Oct 30 13:16:23.103207 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:23.103164 7010 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e792233_9490_4134_acce_5584ab43123a.slice/crio-2a4854682bc850060336bd6497bf4f91fa6dce2a90552b13d418c0f0852fe925\": RecentStats: unable to find data in memory cache]" Oct 30 13:16:29.408577 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:29.408538 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:16:39.063693 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:39.063613 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54"] Oct 30 13:16:39.064124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:39.063829 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" podUID="000e5172-d652-4668-b6b7-cb8ee3f14ad9" containerName="istio-proxy" containerID="cri-o://32aa77ef006c852f3813c2383ec0f531cbe5823145fde38d8f8d9959ad815d33" gracePeriod=30 Oct 30 13:16:39.076788 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:39.076758 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk"] Oct 30 13:16:39.077033 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:39.076979 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" podUID="0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" containerName="istio-proxy" containerID="cri-o://67f3ef67bd346d3b3fd1f496ce7b388bc9e9e387ce5658a551b37c808558764d" gracePeriod=30 Oct 30 13:16:39.793038 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:39.792999 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk"] Oct 30 13:16:39.793376 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:39.793344 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" podUID="22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" containerName="istio-proxy" containerID="cri-o://eb729006c3e68070423ca8715cd0e6998f17502f79ae4a324a07b20ade0287aa" gracePeriod=30 Oct 30 13:16:40.153452 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:40.153407 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb"] Oct 30 13:16:40.153890 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:40.153646 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" podUID="102a82c8-155f-4927-baa7-ae53a3b0fee1" containerName="istio-proxy" containerID="cri-o://5d579517abb1dd817fa70828bca205c4c388a3e2f404d2d63d85c4e1709051f0" gracePeriod=30 Oct 30 13:16:40.252938 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:40.252906 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj"] Oct 30 13:16:40.253116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:40.253096 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" podUID="2f58b078-73f0-4d86-a905-2a914d54687c" containerName="istio-proxy" containerID="cri-o://2044964a3702712a3ba59017befb51dd5b335c6c0f6ad6fd142c27a6884744ef" gracePeriod=30 Oct 30 13:16:40.262858 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:40.262825 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7"] Oct 30 13:16:40.263061 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:40.263038 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" podUID="355db35c-0262-415b-8df0-bede943852c7" containerName="istio-proxy" containerID="cri-o://16751400b853e0df61780310f1d8c68d39bb65bf594b1ad73d0c1708cedbd106" gracePeriod=30 Oct 30 13:16:40.572738 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:40.572639 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8"] Oct 30 13:16:40.574087 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:40.573611 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" podUID="3bcbfc27-09bc-41ec-b7f7-8320b5e56172" containerName="istio-proxy" containerID="cri-o://4c770b2cd933f1a6bba22b9db989625dcdb33aa3696018576bb782c72d19bc47" gracePeriod=30 Oct 30 13:16:44.393400 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.393365 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:16:44.422134 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.422112 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:16:44.495384 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.495343 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-workload-certs\") pod \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " Oct 30 13:16:44.495620 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.495425 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-credential-socket\") pod \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " Oct 30 13:16:44.495620 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.495478 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-podinfo\") pod \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " Oct 30 13:16:44.495620 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.495555 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-token\" (UniqueName: \"kubernetes.io/projected/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-token\") pod \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " Oct 30 13:16:44.495800 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.495665 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-workload-certs" pod "000e5172-d652-4668-b6b7-cb8ee3f14ad9" (UID: "000e5172-d652-4668-b6b7-cb8ee3f14ad9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:44.495983 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.495952 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-kube-api-access-s4852\" (UniqueName: \"kubernetes.io/projected/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-kube-api-access-s4852\") pod \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " Oct 30 13:16:44.496146 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.496019 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-workload-certs\") pod \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " Oct 30 13:16:44.496146 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.496122 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-credential-socket" pod "000e5172-d652-4668-b6b7-cb8ee3f14ad9" (UID: "000e5172-d652-4668-b6b7-cb8ee3f14ad9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:44.496362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.496316 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-workload-certs" pod "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" (UID: "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:44.496467 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.496448 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istiod-ca-cert\") pod \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " Oct 30 13:16:44.496565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.496524 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-data\") pod \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " Oct 30 13:16:44.496629 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.496592 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-podinfo\") pod \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " Oct 30 13:16:44.496836 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.496813 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-data" pod "000e5172-d652-4668-b6b7-cb8ee3f14ad9" (UID: "000e5172-d652-4668-b6b7-cb8ee3f14ad9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:44.496905 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.496834 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istiod-ca-cert" pod "000e5172-d652-4668-b6b7-cb8ee3f14ad9" (UID: "000e5172-d652-4668-b6b7-cb8ee3f14ad9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:44.496905 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.496893 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-workload-socket\") pod \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " Oct 30 13:16:44.497016 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.496942 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-workload-socket\") pod \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " Oct 30 13:16:44.497016 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.496986 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/000e5172-d652-4668-b6b7-cb8ee3f14ad9-kube-api-access-bqjzb\" (UniqueName: \"kubernetes.io/projected/000e5172-d652-4668-b6b7-cb8ee3f14ad9-kube-api-access-bqjzb\") pod \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " Oct 30 13:16:44.497162 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.497141 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-workload-socket" pod "000e5172-d652-4668-b6b7-cb8ee3f14ad9" (UID: "000e5172-d652-4668-b6b7-cb8ee3f14ad9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:44.497222 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.497056 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-data\") pod \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " Oct 30 13:16:44.497222 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.497198 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istiod-ca-cert\") pod \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " Oct 30 13:16:44.497372 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.497353 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-credential-socket\") pod \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " Oct 30 13:16:44.497462 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.497446 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-envoy\") pod \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\" (UID: \"000e5172-d652-4668-b6b7-cb8ee3f14ad9\") " Oct 30 13:16:44.497609 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.497575 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-token\") pod \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " Oct 30 13:16:44.497674 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.497635 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-envoy\") pod \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\" (UID: \"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9\") " Oct 30 13:16:44.498279 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.498250 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-kube-api-access-s4852" pod "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" (UID: "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9"). InnerVolumeSpecName "kube-api-access-s4852". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:44.498406 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.498380 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-podinfo" pod "000e5172-d652-4668-b6b7-cb8ee3f14ad9" (UID: "000e5172-d652-4668-b6b7-cb8ee3f14ad9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:44.498473 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.498402 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.498473 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.498421 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.498473 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.498436 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4852\" (UniqueName: \"kubernetes.io/projected/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-kube-api-access-s4852\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.498473 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.498452 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.498473 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.498468 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.498777 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.498482 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.498777 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.498568 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.498777 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.498589 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-data" pod "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" (UID: "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:44.498777 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.498662 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-credential-socket" pod "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" (UID: "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:44.498976 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.498872 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istiod-ca-cert" pod "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" (UID: "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:44.499416 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.499389 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-workload-socket" pod "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" (UID: "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:44.500518 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.500465 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-token" pod "000e5172-d652-4668-b6b7-cb8ee3f14ad9" (UID: "000e5172-d652-4668-b6b7-cb8ee3f14ad9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:44.500743 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.500699 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-podinfo" pod "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" (UID: "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:44.500900 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.500869 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-token" pod "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" (UID: "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:44.506246 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.501721 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000e5172-d652-4668-b6b7-cb8ee3f14ad9-kube-api-access-bqjzb" pod "000e5172-d652-4668-b6b7-cb8ee3f14ad9" (UID: "000e5172-d652-4668-b6b7-cb8ee3f14ad9"). InnerVolumeSpecName "kube-api-access-bqjzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:44.506246 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.502906 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-envoy" pod "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" (UID: "0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:44.506246 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.504764 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-envoy" pod "000e5172-d652-4668-b6b7-cb8ee3f14ad9" (UID: "000e5172-d652-4668-b6b7-cb8ee3f14ad9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:44.556140 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.556051 7010 generic.go:356] "Generic (PLEG): container finished" podID="000e5172-d652-4668-b6b7-cb8ee3f14ad9" containerID="32aa77ef006c852f3813c2383ec0f531cbe5823145fde38d8f8d9959ad815d33" exitCode=0 Oct 30 13:16:44.556140 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.556116 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" Oct 30 13:16:44.556410 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.556179 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" event={"ID":"000e5172-d652-4668-b6b7-cb8ee3f14ad9","Type":"ContainerDied","Data":"32aa77ef006c852f3813c2383ec0f531cbe5823145fde38d8f8d9959ad815d33"} Oct 30 13:16:44.556410 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.556236 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54" event={"ID":"000e5172-d652-4668-b6b7-cb8ee3f14ad9","Type":"ContainerDied","Data":"a9d341b610f58f5c3208d080edbf7fc36b40ebdf16e31a87bdf21a89e856bf5a"} Oct 30 13:16:44.556410 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.556270 7010 scope.go:122] "RemoveContainer" containerID="32aa77ef006c852f3813c2383ec0f531cbe5823145fde38d8f8d9959ad815d33" Oct 30 13:16:44.558298 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.558256 7010 generic.go:356] "Generic (PLEG): container finished" podID="0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" containerID="67f3ef67bd346d3b3fd1f496ce7b388bc9e9e387ce5658a551b37c808558764d" exitCode=0 Oct 30 13:16:44.558425 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.558330 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" event={"ID":"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9","Type":"ContainerDied","Data":"67f3ef67bd346d3b3fd1f496ce7b388bc9e9e387ce5658a551b37c808558764d"} Oct 30 13:16:44.558425 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.558375 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" event={"ID":"0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9","Type":"ContainerDied","Data":"34ee4c6883686224b1377a05b75241fa052fae4dc5fe599d31bd9be0ccba7ba6"} Oct 30 13:16:44.558612 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.558447 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk" Oct 30 13:16:44.568208 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.568180 7010 scope.go:122] "RemoveContainer" containerID="32aa77ef006c852f3813c2383ec0f531cbe5823145fde38d8f8d9959ad815d33" Oct 30 13:16:44.568577 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:44.568552 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32aa77ef006c852f3813c2383ec0f531cbe5823145fde38d8f8d9959ad815d33\": container with ID starting with 32aa77ef006c852f3813c2383ec0f531cbe5823145fde38d8f8d9959ad815d33 not found: ID does not exist" containerID="32aa77ef006c852f3813c2383ec0f531cbe5823145fde38d8f8d9959ad815d33" Oct 30 13:16:44.568689 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.568585 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32aa77ef006c852f3813c2383ec0f531cbe5823145fde38d8f8d9959ad815d33"} err="failed to get container status \"32aa77ef006c852f3813c2383ec0f531cbe5823145fde38d8f8d9959ad815d33\": rpc error: code = NotFound desc = could not find container \"32aa77ef006c852f3813c2383ec0f531cbe5823145fde38d8f8d9959ad815d33\": container with ID starting with 32aa77ef006c852f3813c2383ec0f531cbe5823145fde38d8f8d9959ad815d33 not found: ID does not exist" Oct 30 13:16:44.568689 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.568602 7010 scope.go:122] "RemoveContainer" containerID="67f3ef67bd346d3b3fd1f496ce7b388bc9e9e387ce5658a551b37c808558764d" Oct 30 13:16:44.579354 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.579334 7010 scope.go:122] "RemoveContainer" containerID="67f3ef67bd346d3b3fd1f496ce7b388bc9e9e387ce5658a551b37c808558764d" Oct 30 13:16:44.579676 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:44.579658 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f3ef67bd346d3b3fd1f496ce7b388bc9e9e387ce5658a551b37c808558764d\": container with ID starting with 67f3ef67bd346d3b3fd1f496ce7b388bc9e9e387ce5658a551b37c808558764d not found: ID does not exist" containerID="67f3ef67bd346d3b3fd1f496ce7b388bc9e9e387ce5658a551b37c808558764d" Oct 30 13:16:44.579737 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.579686 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f3ef67bd346d3b3fd1f496ce7b388bc9e9e387ce5658a551b37c808558764d"} err="failed to get container status \"67f3ef67bd346d3b3fd1f496ce7b388bc9e9e387ce5658a551b37c808558764d\": rpc error: code = NotFound desc = could not find container \"67f3ef67bd346d3b3fd1f496ce7b388bc9e9e387ce5658a551b37c808558764d\": container with ID starting with 67f3ef67bd346d3b3fd1f496ce7b388bc9e9e387ce5658a551b37c808558764d not found: ID does not exist" Oct 30 13:16:44.600007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.599946 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.600007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.599968 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.600007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.599977 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.600007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.599986 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.600007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.599996 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.600007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.600004 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.600007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.600011 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.600308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.600020 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/000e5172-d652-4668-b6b7-cb8ee3f14ad9-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.600308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.600028 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.600308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.600037 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.600308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.600045 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqjzb\" (UniqueName: \"kubernetes.io/projected/000e5172-d652-4668-b6b7-cb8ee3f14ad9-kube-api-access-bqjzb\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:44.623095 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.623059 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54"] Oct 30 13:16:44.630371 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.630337 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-zrs54"] Oct 30 13:16:44.683705 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.671439 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk"] Oct 30 13:16:44.686064 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:44.685995 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-wzdgk"] Oct 30 13:16:45.498028 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.497679 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:16:45.499829 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.499796 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="000e5172-d652-4668-b6b7-cb8ee3f14ad9" path="/var/lib/kubelet/pods/000e5172-d652-4668-b6b7-cb8ee3f14ad9/volumes" Oct 30 13:16:45.500649 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.500627 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" path="/var/lib/kubelet/pods/0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9/volumes" Oct 30 13:16:45.526153 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.526125 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/102a82c8-155f-4927-baa7-ae53a3b0fee1-kube-api-access-7dq8k\" (UniqueName: \"kubernetes.io/projected/102a82c8-155f-4927-baa7-ae53a3b0fee1-kube-api-access-7dq8k\") pod \"102a82c8-155f-4927-baa7-ae53a3b0fee1\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " Oct 30 13:16:45.526287 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.526170 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/102a82c8-155f-4927-baa7-ae53a3b0fee1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/102a82c8-155f-4927-baa7-ae53a3b0fee1-istiod-ca-cert\") pod \"102a82c8-155f-4927-baa7-ae53a3b0fee1\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " Oct 30 13:16:45.526621 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.526593 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102a82c8-155f-4927-baa7-ae53a3b0fee1-istiod-ca-cert" pod "102a82c8-155f-4927-baa7-ae53a3b0fee1" (UID: "102a82c8-155f-4927-baa7-ae53a3b0fee1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:45.526690 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.526231 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-workload-certs\") pod \"102a82c8-155f-4927-baa7-ae53a3b0fee1\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " Oct 30 13:16:45.526954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.526932 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-workload-certs" pod "102a82c8-155f-4927-baa7-ae53a3b0fee1" (UID: "102a82c8-155f-4927-baa7-ae53a3b0fee1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.527063 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.527043 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-token\" (UniqueName: \"kubernetes.io/projected/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-token\") pod \"102a82c8-155f-4927-baa7-ae53a3b0fee1\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " Oct 30 13:16:45.527623 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.527601 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-podinfo\") pod \"102a82c8-155f-4927-baa7-ae53a3b0fee1\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " Oct 30 13:16:45.528147 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.528130 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-envoy\") pod \"102a82c8-155f-4927-baa7-ae53a3b0fee1\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " Oct 30 13:16:45.528650 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.528619 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-data\") pod \"102a82c8-155f-4927-baa7-ae53a3b0fee1\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " Oct 30 13:16:45.529176 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.529055 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-data" pod "102a82c8-155f-4927-baa7-ae53a3b0fee1" (UID: "102a82c8-155f-4927-baa7-ae53a3b0fee1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.529322 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.529305 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-credential-socket\") pod \"102a82c8-155f-4927-baa7-ae53a3b0fee1\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " Oct 30 13:16:45.529870 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.529674 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-credential-socket" pod "102a82c8-155f-4927-baa7-ae53a3b0fee1" (UID: "102a82c8-155f-4927-baa7-ae53a3b0fee1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.530067 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.530050 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-workload-socket\") pod \"102a82c8-155f-4927-baa7-ae53a3b0fee1\" (UID: \"102a82c8-155f-4927-baa7-ae53a3b0fee1\") " Oct 30 13:16:45.530524 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.530356 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-workload-socket" pod "102a82c8-155f-4927-baa7-ae53a3b0fee1" (UID: "102a82c8-155f-4927-baa7-ae53a3b0fee1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.531242 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.531216 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.531343 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.531280 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.531343 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.531296 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.531343 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.531310 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/102a82c8-155f-4927-baa7-ae53a3b0fee1-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.531343 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.531324 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.534900 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.534875 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-envoy" pod "102a82c8-155f-4927-baa7-ae53a3b0fee1" (UID: "102a82c8-155f-4927-baa7-ae53a3b0fee1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.536901 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.536869 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-token" pod "102a82c8-155f-4927-baa7-ae53a3b0fee1" (UID: "102a82c8-155f-4927-baa7-ae53a3b0fee1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:45.539853 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.539818 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-podinfo" pod "102a82c8-155f-4927-baa7-ae53a3b0fee1" (UID: "102a82c8-155f-4927-baa7-ae53a3b0fee1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:45.543057 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.543020 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102a82c8-155f-4927-baa7-ae53a3b0fee1-kube-api-access-7dq8k" pod "102a82c8-155f-4927-baa7-ae53a3b0fee1" (UID: "102a82c8-155f-4927-baa7-ae53a3b0fee1"). InnerVolumeSpecName "kube-api-access-7dq8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:45.578952 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.578918 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:16:45.582462 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.582256 7010 generic.go:356] "Generic (PLEG): container finished" podID="355db35c-0262-415b-8df0-bede943852c7" containerID="16751400b853e0df61780310f1d8c68d39bb65bf594b1ad73d0c1708cedbd106" exitCode=0 Oct 30 13:16:45.582462 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.582403 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" event={"ID":"355db35c-0262-415b-8df0-bede943852c7","Type":"ContainerDied","Data":"16751400b853e0df61780310f1d8c68d39bb65bf594b1ad73d0c1708cedbd106"} Oct 30 13:16:45.584524 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.584315 7010 generic.go:356] "Generic (PLEG): container finished" podID="102a82c8-155f-4927-baa7-ae53a3b0fee1" containerID="5d579517abb1dd817fa70828bca205c4c388a3e2f404d2d63d85c4e1709051f0" exitCode=0 Oct 30 13:16:45.584524 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.584402 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" event={"ID":"102a82c8-155f-4927-baa7-ae53a3b0fee1","Type":"ContainerDied","Data":"5d579517abb1dd817fa70828bca205c4c388a3e2f404d2d63d85c4e1709051f0"} Oct 30 13:16:45.584524 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.584442 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" event={"ID":"102a82c8-155f-4927-baa7-ae53a3b0fee1","Type":"ContainerDied","Data":"1b08e839e9cf2f4fc137e81fca97f13e51ae3a4d88c55a1fe6d5685387c157be"} Oct 30 13:16:45.584524 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.584472 7010 scope.go:122] "RemoveContainer" containerID="5d579517abb1dd817fa70828bca205c4c388a3e2f404d2d63d85c4e1709051f0" Oct 30 13:16:45.588523 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.585028 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb" Oct 30 13:16:45.595537 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.594720 7010 generic.go:356] "Generic (PLEG): container finished" podID="2f58b078-73f0-4d86-a905-2a914d54687c" containerID="2044964a3702712a3ba59017befb51dd5b335c6c0f6ad6fd142c27a6884744ef" exitCode=0 Oct 30 13:16:45.595537 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.594823 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" event={"ID":"2f58b078-73f0-4d86-a905-2a914d54687c","Type":"ContainerDied","Data":"2044964a3702712a3ba59017befb51dd5b335c6c0f6ad6fd142c27a6884744ef"} Oct 30 13:16:45.598842 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.598708 7010 generic.go:356] "Generic (PLEG): container finished" podID="22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" containerID="eb729006c3e68070423ca8715cd0e6998f17502f79ae4a324a07b20ade0287aa" exitCode=0 Oct 30 13:16:45.598945 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.598797 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" event={"ID":"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81","Type":"ContainerDied","Data":"eb729006c3e68070423ca8715cd0e6998f17502f79ae4a324a07b20ade0287aa"} Oct 30 13:16:45.600198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.599094 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" event={"ID":"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81","Type":"ContainerDied","Data":"abd3c6aa98c4eb5c11a033c041e27aecd65562c6c26b9cc68fe3aebcd1f07c32"} Oct 30 13:16:45.600198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.599158 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk" Oct 30 13:16:45.617720 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.617697 7010 scope.go:122] "RemoveContainer" containerID="5d579517abb1dd817fa70828bca205c4c388a3e2f404d2d63d85c4e1709051f0" Oct 30 13:16:45.618271 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:45.618108 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d579517abb1dd817fa70828bca205c4c388a3e2f404d2d63d85c4e1709051f0\": container with ID starting with 5d579517abb1dd817fa70828bca205c4c388a3e2f404d2d63d85c4e1709051f0 not found: ID does not exist" containerID="5d579517abb1dd817fa70828bca205c4c388a3e2f404d2d63d85c4e1709051f0" Oct 30 13:16:45.618271 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.618150 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d579517abb1dd817fa70828bca205c4c388a3e2f404d2d63d85c4e1709051f0"} err="failed to get container status \"5d579517abb1dd817fa70828bca205c4c388a3e2f404d2d63d85c4e1709051f0\": rpc error: code = NotFound desc = could not find container \"5d579517abb1dd817fa70828bca205c4c388a3e2f404d2d63d85c4e1709051f0\": container with ID starting with 5d579517abb1dd817fa70828bca205c4c388a3e2f404d2d63d85c4e1709051f0 not found: ID does not exist" Oct 30 13:16:45.618271 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.618178 7010 scope.go:122] "RemoveContainer" containerID="eb729006c3e68070423ca8715cd0e6998f17502f79ae4a324a07b20ade0287aa" Oct 30 13:16:45.632806 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.632779 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-kube-api-access-cft8q\" (UniqueName: \"kubernetes.io/projected/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-kube-api-access-cft8q\") pod \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " Oct 30 13:16:45.632939 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.632848 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-envoy\") pod \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " Oct 30 13:16:45.633004 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.632965 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-data\") pod \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " Oct 30 13:16:45.633059 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.633007 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istiod-ca-cert\") pod \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " Oct 30 13:16:45.633867 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.633072 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-podinfo\") pod \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " Oct 30 13:16:45.633867 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.633725 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istiod-ca-cert" pod "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" (UID: "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:45.634000 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.633896 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-data" pod "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" (UID: "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.634000 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.633970 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-workload-certs\") pod \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " Oct 30 13:16:45.634092 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.634017 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-workload-socket\") pod \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " Oct 30 13:16:45.634245 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.634224 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-workload-certs" pod "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" (UID: "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.634370 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.634333 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-workload-socket" pod "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" (UID: "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.634675 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.634657 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-token\" (UniqueName: \"kubernetes.io/projected/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-token\") pod \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " Oct 30 13:16:45.634855 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.634837 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-credential-socket\") pod \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\" (UID: \"22f40d0a-c583-4fd0-bed7-fcfa1b0abd81\") " Oct 30 13:16:45.635105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.635078 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-credential-socket" pod "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" (UID: "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.635974 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.635947 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.636128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.635979 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.636128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.635995 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.636128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.636008 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.636128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.636021 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.636128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.636034 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.636128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.636048 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/102a82c8-155f-4927-baa7-ae53a3b0fee1-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.636128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.636062 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.636128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.636075 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dq8k\" (UniqueName: \"kubernetes.io/projected/102a82c8-155f-4927-baa7-ae53a3b0fee1-kube-api-access-7dq8k\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.640216 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.639971 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-token" pod "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" (UID: "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:45.651403 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.651358 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-podinfo" pod "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" (UID: "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:45.651578 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.651512 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-kube-api-access-cft8q" pod "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" (UID: "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81"). InnerVolumeSpecName "kube-api-access-cft8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:45.652056 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.652028 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-envoy" pod "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" (UID: "22f40d0a-c583-4fd0-bed7-fcfa1b0abd81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.677187 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.677159 7010 scope.go:122] "RemoveContainer" containerID="eb729006c3e68070423ca8715cd0e6998f17502f79ae4a324a07b20ade0287aa" Oct 30 13:16:45.677680 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:45.677655 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb729006c3e68070423ca8715cd0e6998f17502f79ae4a324a07b20ade0287aa\": container with ID starting with eb729006c3e68070423ca8715cd0e6998f17502f79ae4a324a07b20ade0287aa not found: ID does not exist" containerID="eb729006c3e68070423ca8715cd0e6998f17502f79ae4a324a07b20ade0287aa" Oct 30 13:16:45.677764 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.677694 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb729006c3e68070423ca8715cd0e6998f17502f79ae4a324a07b20ade0287aa"} err="failed to get container status \"eb729006c3e68070423ca8715cd0e6998f17502f79ae4a324a07b20ade0287aa\": rpc error: code = NotFound desc = could not find container \"eb729006c3e68070423ca8715cd0e6998f17502f79ae4a324a07b20ade0287aa\": container with ID starting with eb729006c3e68070423ca8715cd0e6998f17502f79ae4a324a07b20ade0287aa not found: ID does not exist" Oct 30 13:16:45.712640 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.712607 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb"] Oct 30 13:16:45.716725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.716613 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:16:45.722028 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.722000 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-7d9pb"] Oct 30 13:16:45.724784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.724764 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:16:45.738244 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.738178 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.738244 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.738216 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cft8q\" (UniqueName: \"kubernetes.io/projected/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-kube-api-access-cft8q\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.738244 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.738234 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.738244 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.738249 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.839063 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.839006 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-workload-socket\") pod \"2f58b078-73f0-4d86-a905-2a914d54687c\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " Oct 30 13:16:45.839240 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.839176 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/355db35c-0262-415b-8df0-bede943852c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/355db35c-0262-415b-8df0-bede943852c7-istio-podinfo\") pod \"355db35c-0262-415b-8df0-bede943852c7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " Oct 30 13:16:45.839240 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.839219 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-istio-envoy\") pod \"355db35c-0262-415b-8df0-bede943852c7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " Oct 30 13:16:45.839346 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.839281 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-istio-data\") pod \"2f58b078-73f0-4d86-a905-2a914d54687c\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " Oct 30 13:16:45.839346 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.839325 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f58b078-73f0-4d86-a905-2a914d54687c-istio-token\" (UniqueName: \"kubernetes.io/projected/2f58b078-73f0-4d86-a905-2a914d54687c-istio-token\") pod \"2f58b078-73f0-4d86-a905-2a914d54687c\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " Oct 30 13:16:45.839445 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.839386 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355db35c-0262-415b-8df0-bede943852c7-kube-api-access-c2rjq\" (UniqueName: \"kubernetes.io/projected/355db35c-0262-415b-8df0-bede943852c7-kube-api-access-c2rjq\") pod \"355db35c-0262-415b-8df0-bede943852c7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " Oct 30 13:16:45.839788 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.839453 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-workload-socket\") pod \"355db35c-0262-415b-8df0-bede943852c7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " Oct 30 13:16:45.839788 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.839519 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-workload-certs\") pod \"355db35c-0262-415b-8df0-bede943852c7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " Oct 30 13:16:45.839788 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.839611 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f58b078-73f0-4d86-a905-2a914d54687c-kube-api-access-5hbwt\" (UniqueName: \"kubernetes.io/projected/2f58b078-73f0-4d86-a905-2a914d54687c-kube-api-access-5hbwt\") pod \"2f58b078-73f0-4d86-a905-2a914d54687c\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " Oct 30 13:16:45.839969 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.839786 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-istio-envoy\") pod \"2f58b078-73f0-4d86-a905-2a914d54687c\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " Oct 30 13:16:45.839969 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.839856 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-workload-certs\") pod \"2f58b078-73f0-4d86-a905-2a914d54687c\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " Oct 30 13:16:45.839969 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.839887 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f58b078-73f0-4d86-a905-2a914d54687c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f58b078-73f0-4d86-a905-2a914d54687c-istiod-ca-cert\") pod \"2f58b078-73f0-4d86-a905-2a914d54687c\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " Oct 30 13:16:45.840124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.839960 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-istio-data\") pod \"355db35c-0262-415b-8df0-bede943852c7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " Oct 30 13:16:45.840124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.840008 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355db35c-0262-415b-8df0-bede943852c7-istio-token\" (UniqueName: \"kubernetes.io/projected/355db35c-0262-415b-8df0-bede943852c7-istio-token\") pod \"355db35c-0262-415b-8df0-bede943852c7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " Oct 30 13:16:45.840124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.840069 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-credential-socket\") pod \"355db35c-0262-415b-8df0-bede943852c7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " Oct 30 13:16:45.840281 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.840152 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/355db35c-0262-415b-8df0-bede943852c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/355db35c-0262-415b-8df0-bede943852c7-istiod-ca-cert\") pod \"355db35c-0262-415b-8df0-bede943852c7\" (UID: \"355db35c-0262-415b-8df0-bede943852c7\") " Oct 30 13:16:45.840281 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.840230 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-credential-socket\") pod \"2f58b078-73f0-4d86-a905-2a914d54687c\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " Oct 30 13:16:45.840377 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.840287 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f58b078-73f0-4d86-a905-2a914d54687c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f58b078-73f0-4d86-a905-2a914d54687c-istio-podinfo\") pod \"2f58b078-73f0-4d86-a905-2a914d54687c\" (UID: \"2f58b078-73f0-4d86-a905-2a914d54687c\") " Oct 30 13:16:45.840725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.840701 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-istio-data" pod "355db35c-0262-415b-8df0-bede943852c7" (UID: "355db35c-0262-415b-8df0-bede943852c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.841681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.841041 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-istio-data" pod "2f58b078-73f0-4d86-a905-2a914d54687c" (UID: "2f58b078-73f0-4d86-a905-2a914d54687c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.841681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.841515 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.841681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.841541 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.841681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.841543 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-workload-socket" pod "355db35c-0262-415b-8df0-bede943852c7" (UID: "355db35c-0262-415b-8df0-bede943852c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.841983 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.841702 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-workload-socket" pod "2f58b078-73f0-4d86-a905-2a914d54687c" (UID: "2f58b078-73f0-4d86-a905-2a914d54687c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.841983 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.841831 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-credential-socket" pod "2f58b078-73f0-4d86-a905-2a914d54687c" (UID: "2f58b078-73f0-4d86-a905-2a914d54687c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.841983 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.841882 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-workload-certs" pod "355db35c-0262-415b-8df0-bede943852c7" (UID: "355db35c-0262-415b-8df0-bede943852c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.841983 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.841901 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/355db35c-0262-415b-8df0-bede943852c7-istiod-ca-cert" pod "355db35c-0262-415b-8df0-bede943852c7" (UID: "355db35c-0262-415b-8df0-bede943852c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:45.841983 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.841944 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-credential-socket" pod "355db35c-0262-415b-8df0-bede943852c7" (UID: "355db35c-0262-415b-8df0-bede943852c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.842405 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.842031 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f58b078-73f0-4d86-a905-2a914d54687c-istiod-ca-cert" pod "2f58b078-73f0-4d86-a905-2a914d54687c" (UID: "2f58b078-73f0-4d86-a905-2a914d54687c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:45.844297 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.844269 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-workload-certs" pod "2f58b078-73f0-4d86-a905-2a914d54687c" (UID: "2f58b078-73f0-4d86-a905-2a914d54687c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.846162 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.846138 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-istio-envoy" pod "2f58b078-73f0-4d86-a905-2a914d54687c" (UID: "2f58b078-73f0-4d86-a905-2a914d54687c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.846563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.846521 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f58b078-73f0-4d86-a905-2a914d54687c-istio-podinfo" pod "2f58b078-73f0-4d86-a905-2a914d54687c" (UID: "2f58b078-73f0-4d86-a905-2a914d54687c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:45.850765 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.850738 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/355db35c-0262-415b-8df0-bede943852c7-istio-podinfo" pod "355db35c-0262-415b-8df0-bede943852c7" (UID: "355db35c-0262-415b-8df0-bede943852c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:45.851542 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.851502 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-istio-envoy" pod "355db35c-0262-415b-8df0-bede943852c7" (UID: "355db35c-0262-415b-8df0-bede943852c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:45.851656 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.851556 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355db35c-0262-415b-8df0-bede943852c7-kube-api-access-c2rjq" pod "355db35c-0262-415b-8df0-bede943852c7" (UID: "355db35c-0262-415b-8df0-bede943852c7"). InnerVolumeSpecName "kube-api-access-c2rjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:45.851846 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.851815 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f58b078-73f0-4d86-a905-2a914d54687c-kube-api-access-5hbwt" pod "2f58b078-73f0-4d86-a905-2a914d54687c" (UID: "2f58b078-73f0-4d86-a905-2a914d54687c"). InnerVolumeSpecName "kube-api-access-5hbwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:45.852084 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.852061 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f58b078-73f0-4d86-a905-2a914d54687c-istio-token" pod "2f58b078-73f0-4d86-a905-2a914d54687c" (UID: "2f58b078-73f0-4d86-a905-2a914d54687c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:45.853276 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.853254 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355db35c-0262-415b-8df0-bede943852c7-istio-token" pod "355db35c-0262-415b-8df0-bede943852c7" (UID: "355db35c-0262-415b-8df0-bede943852c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:45.922611 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.922581 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:16:45.942836 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.942797 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.942836 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.942830 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/355db35c-0262-415b-8df0-bede943852c7-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.942836 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.942838 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.942836 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.942847 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f58b078-73f0-4d86-a905-2a914d54687c-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.943104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.942857 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2rjq\" (UniqueName: \"kubernetes.io/projected/355db35c-0262-415b-8df0-bede943852c7-kube-api-access-c2rjq\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.943104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.942866 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.943104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.942876 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.943104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.942888 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hbwt\" (UniqueName: \"kubernetes.io/projected/2f58b078-73f0-4d86-a905-2a914d54687c-kube-api-access-5hbwt\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.943104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.942896 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.943104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.942904 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.943104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.942912 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f58b078-73f0-4d86-a905-2a914d54687c-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.943104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.942920 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/355db35c-0262-415b-8df0-bede943852c7-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.943104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.942927 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/355db35c-0262-415b-8df0-bede943852c7-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.943104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.942979 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/355db35c-0262-415b-8df0-bede943852c7-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.943104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.942988 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f58b078-73f0-4d86-a905-2a914d54687c-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.943104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.942996 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f58b078-73f0-4d86-a905-2a914d54687c-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:45.950079 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.950029 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk"] Oct 30 13:16:45.953528 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:45.953482 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-pkfgk"] Oct 30 13:16:46.043955 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.043917 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istiod-ca-cert\") pod \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " Oct 30 13:16:46.044174 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.044013 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-token\" (UniqueName: \"kubernetes.io/projected/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-token\") pod \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " Oct 30 13:16:46.044174 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.044050 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-workload-certs\") pod \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " Oct 30 13:16:46.044174 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.044097 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-workload-socket\") pod \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " Oct 30 13:16:46.044355 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.044202 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-envoy\") pod \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " Oct 30 13:16:46.044355 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.044260 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-kube-api-access-4lgz9\" (UniqueName: \"kubernetes.io/projected/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-kube-api-access-4lgz9\") pod \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " Oct 30 13:16:46.044463 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.044433 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-podinfo\") pod \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " Oct 30 13:16:46.044558 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.044482 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-credential-socket\") pod \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " Oct 30 13:16:46.045557 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.044998 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-credential-socket" pod "3bcbfc27-09bc-41ec-b7f7-8320b5e56172" (UID: "3bcbfc27-09bc-41ec-b7f7-8320b5e56172"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:46.045557 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.045375 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-workload-certs" pod "3bcbfc27-09bc-41ec-b7f7-8320b5e56172" (UID: "3bcbfc27-09bc-41ec-b7f7-8320b5e56172"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:46.045557 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.045457 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istiod-ca-cert" pod "3bcbfc27-09bc-41ec-b7f7-8320b5e56172" (UID: "3bcbfc27-09bc-41ec-b7f7-8320b5e56172"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:46.045820 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.045671 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-data" pod "3bcbfc27-09bc-41ec-b7f7-8320b5e56172" (UID: "3bcbfc27-09bc-41ec-b7f7-8320b5e56172"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:46.046772 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.045924 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-data\") pod \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\" (UID: \"3bcbfc27-09bc-41ec-b7f7-8320b5e56172\") " Oct 30 13:16:46.046772 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.046024 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-workload-socket" pod "3bcbfc27-09bc-41ec-b7f7-8320b5e56172" (UID: "3bcbfc27-09bc-41ec-b7f7-8320b5e56172"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:46.046772 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.046686 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.046772 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.046709 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.046772 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.046723 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.046772 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.046736 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.046772 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.046748 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.049543 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.049477 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-kube-api-access-4lgz9" pod "3bcbfc27-09bc-41ec-b7f7-8320b5e56172" (UID: "3bcbfc27-09bc-41ec-b7f7-8320b5e56172"). InnerVolumeSpecName "kube-api-access-4lgz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:46.050411 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.050376 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-envoy" pod "3bcbfc27-09bc-41ec-b7f7-8320b5e56172" (UID: "3bcbfc27-09bc-41ec-b7f7-8320b5e56172"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:46.050832 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.050807 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-token" pod "3bcbfc27-09bc-41ec-b7f7-8320b5e56172" (UID: "3bcbfc27-09bc-41ec-b7f7-8320b5e56172"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:46.052168 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.052142 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-podinfo" pod "3bcbfc27-09bc-41ec-b7f7-8320b5e56172" (UID: "3bcbfc27-09bc-41ec-b7f7-8320b5e56172"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:46.153517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.152933 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.153517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.152969 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.153517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.152982 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lgz9\" (UniqueName: \"kubernetes.io/projected/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-kube-api-access-4lgz9\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.153517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.152997 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bcbfc27-09bc-41ec-b7f7-8320b5e56172-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:46.628247 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.626358 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" event={"ID":"355db35c-0262-415b-8df0-bede943852c7","Type":"ContainerDied","Data":"f8e9c766cc71edaed7e8f6059eb663dd8a789450a905c7ab5120dd56ff4475d3"} Oct 30 13:16:46.628247 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.626437 7010 scope.go:122] "RemoveContainer" containerID="16751400b853e0df61780310f1d8c68d39bb65bf594b1ad73d0c1708cedbd106" Oct 30 13:16:46.628247 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.626575 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7" Oct 30 13:16:46.655795 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.653080 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" Oct 30 13:16:46.655795 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.653086 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" event={"ID":"3bcbfc27-09bc-41ec-b7f7-8320b5e56172","Type":"ContainerDied","Data":"4c770b2cd933f1a6bba22b9db989625dcdb33aa3696018576bb782c72d19bc47"} Oct 30 13:16:46.655795 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.652955 7010 generic.go:356] "Generic (PLEG): container finished" podID="3bcbfc27-09bc-41ec-b7f7-8320b5e56172" containerID="4c770b2cd933f1a6bba22b9db989625dcdb33aa3696018576bb782c72d19bc47" exitCode=0 Oct 30 13:16:46.655795 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.655684 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8" event={"ID":"3bcbfc27-09bc-41ec-b7f7-8320b5e56172","Type":"ContainerDied","Data":"c6403af4f3efe70d534aa9aa02a9cf78f79a3ffd144b6bb2d04e646ea6bf7acd"} Oct 30 13:16:46.662532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.661099 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" event={"ID":"2f58b078-73f0-4d86-a905-2a914d54687c","Type":"ContainerDied","Data":"4fb8c0e09fa544fa8a1ebe96a867bf5c421bef326565c04619acecc47d79f6b9"} Oct 30 13:16:46.662532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.661221 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj" Oct 30 13:16:46.668816 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.668700 7010 scope.go:122] "RemoveContainer" containerID="4c770b2cd933f1a6bba22b9db989625dcdb33aa3696018576bb782c72d19bc47" Oct 30 13:16:46.679783 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.679758 7010 scope.go:122] "RemoveContainer" containerID="4c770b2cd933f1a6bba22b9db989625dcdb33aa3696018576bb782c72d19bc47" Oct 30 13:16:46.680160 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:46.680140 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c770b2cd933f1a6bba22b9db989625dcdb33aa3696018576bb782c72d19bc47\": container with ID starting with 4c770b2cd933f1a6bba22b9db989625dcdb33aa3696018576bb782c72d19bc47 not found: ID does not exist" containerID="4c770b2cd933f1a6bba22b9db989625dcdb33aa3696018576bb782c72d19bc47" Oct 30 13:16:46.680251 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.680172 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c770b2cd933f1a6bba22b9db989625dcdb33aa3696018576bb782c72d19bc47"} err="failed to get container status \"4c770b2cd933f1a6bba22b9db989625dcdb33aa3696018576bb782c72d19bc47\": rpc error: code = NotFound desc = could not find container \"4c770b2cd933f1a6bba22b9db989625dcdb33aa3696018576bb782c72d19bc47\": container with ID starting with 4c770b2cd933f1a6bba22b9db989625dcdb33aa3696018576bb782c72d19bc47 not found: ID does not exist" Oct 30 13:16:46.680251 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.680197 7010 scope.go:122] "RemoveContainer" containerID="2044964a3702712a3ba59017befb51dd5b335c6c0f6ad6fd142c27a6884744ef" Oct 30 13:16:46.716093 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.716022 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7"] Oct 30 13:16:46.727954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.727854 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-r44h7"] Oct 30 13:16:46.816562 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.816526 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8"] Oct 30 13:16:46.823566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.823532 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-94nb8"] Oct 30 13:16:46.883992 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.883905 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj"] Oct 30 13:16:46.905354 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:46.905319 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-vbdsj"] Oct 30 13:16:47.274448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.274368 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824"] Oct 30 13:16:47.275046 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275030 7010 cpu_manager.go:414] "containerMap: removing container" podUID="0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" containerName="istio-proxy" Oct 30 13:16:47.275100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275047 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" containerName="istio-proxy" Oct 30 13:16:47.275100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275072 7010 cpu_manager.go:414] "containerMap: removing container" podUID="d5dacfc7-1c21-4ac5-b93a-1f0763af424f" containerName="istio-proxy" Oct 30 13:16:47.275100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275078 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5dacfc7-1c21-4ac5-b93a-1f0763af424f" containerName="istio-proxy" Oct 30 13:16:47.275100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275090 7010 cpu_manager.go:414] "containerMap: removing container" podUID="22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" containerName="istio-proxy" Oct 30 13:16:47.275100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275096 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" containerName="istio-proxy" Oct 30 13:16:47.275243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275104 7010 cpu_manager.go:414] "containerMap: removing container" podUID="102a82c8-155f-4927-baa7-ae53a3b0fee1" containerName="istio-proxy" Oct 30 13:16:47.275243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275109 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102a82c8-155f-4927-baa7-ae53a3b0fee1" containerName="istio-proxy" Oct 30 13:16:47.275243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275125 7010 cpu_manager.go:414] "containerMap: removing container" podUID="000e5172-d652-4668-b6b7-cb8ee3f14ad9" containerName="istio-proxy" Oct 30 13:16:47.275243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275130 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="000e5172-d652-4668-b6b7-cb8ee3f14ad9" containerName="istio-proxy" Oct 30 13:16:47.275243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275136 7010 cpu_manager.go:414] "containerMap: removing container" podUID="3bcbfc27-09bc-41ec-b7f7-8320b5e56172" containerName="istio-proxy" Oct 30 13:16:47.275243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275141 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bcbfc27-09bc-41ec-b7f7-8320b5e56172" containerName="istio-proxy" Oct 30 13:16:47.275243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275150 7010 cpu_manager.go:414] "containerMap: removing container" podUID="428e36fe-a829-4bee-bcd5-5150d5358411" containerName="istio-proxy" Oct 30 13:16:47.275243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275154 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428e36fe-a829-4bee-bcd5-5150d5358411" containerName="istio-proxy" Oct 30 13:16:47.275243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275165 7010 cpu_manager.go:414] "containerMap: removing container" podUID="355db35c-0262-415b-8df0-bede943852c7" containerName="istio-proxy" Oct 30 13:16:47.275243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275169 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="355db35c-0262-415b-8df0-bede943852c7" containerName="istio-proxy" Oct 30 13:16:47.275243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275185 7010 cpu_manager.go:414] "containerMap: removing container" podUID="a0aace59-a54a-447a-ba91-3ab588b2c2fe" containerName="cleanup" Oct 30 13:16:47.275243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275190 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0aace59-a54a-447a-ba91-3ab588b2c2fe" containerName="cleanup" Oct 30 13:16:47.275243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275196 7010 cpu_manager.go:414] "containerMap: removing container" podUID="2f58b078-73f0-4d86-a905-2a914d54687c" containerName="istio-proxy" Oct 30 13:16:47.275243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275201 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f58b078-73f0-4d86-a905-2a914d54687c" containerName="istio-proxy" Oct 30 13:16:47.275243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275210 7010 cpu_manager.go:414] "containerMap: removing container" podUID="ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" containerName="istio-proxy" Oct 30 13:16:47.275243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275214 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" containerName="istio-proxy" Oct 30 13:16:47.275742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275300 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ede3d71-cc65-4660-b836-6f05f27ea6cc" containerName="cleanup" Oct 30 13:16:47.275742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275308 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f58b078-73f0-4d86-a905-2a914d54687c" containerName="istio-proxy" Oct 30 13:16:47.275742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275314 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1cd6b6-1ebf-41ad-a67d-2a0c30b9d6db" containerName="istio-proxy" Oct 30 13:16:47.275742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275320 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="428e36fe-a829-4bee-bcd5-5150d5358411" containerName="istio-proxy" Oct 30 13:16:47.275742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275325 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d86f05b-db3f-417f-b5ab-395ebcbb4a0e" containerName="cleanup" Oct 30 13:16:47.275742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275333 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2d80f6-c3a5-4e9a-8a6c-159d927a69e9" containerName="istio-proxy" Oct 30 13:16:47.275742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275340 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5dacfc7-1c21-4ac5-b93a-1f0763af424f" containerName="istio-proxy" Oct 30 13:16:47.275742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275347 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" containerName="istio-proxy" Oct 30 13:16:47.275742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275351 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="102a82c8-155f-4927-baa7-ae53a3b0fee1" containerName="istio-proxy" Oct 30 13:16:47.275742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275356 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0aace59-a54a-447a-ba91-3ab588b2c2fe" containerName="cleanup" Oct 30 13:16:47.275742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275362 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="000e5172-d652-4668-b6b7-cb8ee3f14ad9" containerName="istio-proxy" Oct 30 13:16:47.275742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275368 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bcbfc27-09bc-41ec-b7f7-8320b5e56172" containerName="istio-proxy" Oct 30 13:16:47.275742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.275374 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="355db35c-0262-415b-8df0-bede943852c7" containerName="istio-proxy" Oct 30 13:16:47.280759 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.280730 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:16:47.338197 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.338164 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824"] Oct 30 13:16:47.357248 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.357216 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-df6c3661-serving-cert\"" Oct 30 13:16:47.391473 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.391426 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqr7h\" (UniqueName: \"kubernetes.io/projected/8493b4e1-efad-4d0f-bfef-28a37facefdb-kube-api-access-wqr7h\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-hg824\" (UID: \"8493b4e1-efad-4d0f-bfef-28a37facefdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:16:47.391677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.391652 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8493b4e1-efad-4d0f-bfef-28a37facefdb-metrics-certs\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-hg824\" (UID: \"8493b4e1-efad-4d0f-bfef-28a37facefdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:16:47.391824 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.391806 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8493b4e1-efad-4d0f-bfef-28a37facefdb-openshift-service-ca\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-hg824\" (UID: \"8493b4e1-efad-4d0f-bfef-28a37facefdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:16:47.391902 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.391877 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8493b4e1-efad-4d0f-bfef-28a37facefdb-maas-api-tls\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-hg824\" (UID: \"8493b4e1-efad-4d0f-bfef-28a37facefdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:16:47.490872 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.490839 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="102a82c8-155f-4927-baa7-ae53a3b0fee1" path="/var/lib/kubelet/pods/102a82c8-155f-4927-baa7-ae53a3b0fee1/volumes" Oct 30 13:16:47.491440 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.491424 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f40d0a-c583-4fd0-bed7-fcfa1b0abd81" path="/var/lib/kubelet/pods/22f40d0a-c583-4fd0-bed7-fcfa1b0abd81/volumes" Oct 30 13:16:47.491947 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.491932 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f58b078-73f0-4d86-a905-2a914d54687c" path="/var/lib/kubelet/pods/2f58b078-73f0-4d86-a905-2a914d54687c/volumes" Oct 30 13:16:47.492756 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.492740 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="355db35c-0262-415b-8df0-bede943852c7" path="/var/lib/kubelet/pods/355db35c-0262-415b-8df0-bede943852c7/volumes" Oct 30 13:16:47.493202 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.493188 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bcbfc27-09bc-41ec-b7f7-8320b5e56172" path="/var/lib/kubelet/pods/3bcbfc27-09bc-41ec-b7f7-8320b5e56172/volumes" Oct 30 13:16:47.493617 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.493594 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8493b4e1-efad-4d0f-bfef-28a37facefdb-openshift-service-ca\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-hg824\" (UID: \"8493b4e1-efad-4d0f-bfef-28a37facefdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:16:47.493669 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.493641 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8493b4e1-efad-4d0f-bfef-28a37facefdb-maas-api-tls\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-hg824\" (UID: \"8493b4e1-efad-4d0f-bfef-28a37facefdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:16:47.493701 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.493680 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqr7h\" (UniqueName: \"kubernetes.io/projected/8493b4e1-efad-4d0f-bfef-28a37facefdb-kube-api-access-wqr7h\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-hg824\" (UID: \"8493b4e1-efad-4d0f-bfef-28a37facefdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:16:47.493784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.493771 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8493b4e1-efad-4d0f-bfef-28a37facefdb-metrics-certs\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-hg824\" (UID: \"8493b4e1-efad-4d0f-bfef-28a37facefdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:16:47.494620 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.494597 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8493b4e1-efad-4d0f-bfef-28a37facefdb-openshift-service-ca\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-hg824\" (UID: \"8493b4e1-efad-4d0f-bfef-28a37facefdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:16:47.496257 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.496236 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8493b4e1-efad-4d0f-bfef-28a37facefdb-metrics-certs\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-hg824\" (UID: \"8493b4e1-efad-4d0f-bfef-28a37facefdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:16:47.496346 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.496255 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8493b4e1-efad-4d0f-bfef-28a37facefdb-maas-api-tls\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-hg824\" (UID: \"8493b4e1-efad-4d0f-bfef-28a37facefdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:16:47.577709 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.577619 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqr7h\" (UniqueName: \"kubernetes.io/projected/8493b4e1-efad-4d0f-bfef-28a37facefdb-kube-api-access-wqr7h\") pod \"maas-api-e2e-mt-df6c3661-6695779d4-hg824\" (UID: \"8493b4e1-efad-4d0f-bfef-28a37facefdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:16:47.607410 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.607372 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:16:47.941333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:47.941304 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824"] Oct 30 13:16:47.943294 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:16:47.943263 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8493b4e1_efad_4d0f_bfef_28a37facefdb.slice/crio-6955984180a85db807d7ae87a7eb2adec4aa7e38fbd94ba01aeb74fd3932ec47 WatchSource:0}: Error finding container 6955984180a85db807d7ae87a7eb2adec4aa7e38fbd94ba01aeb74fd3932ec47: Status 404 returned error can't find the container with id 6955984180a85db807d7ae87a7eb2adec4aa7e38fbd94ba01aeb74fd3932ec47 Oct 30 13:16:48.708947 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:48.707606 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" event={"ID":"8493b4e1-efad-4d0f-bfef-28a37facefdb","Type":"ContainerStarted","Data":"1d1e9f021945c033b4fdaa3de1748b6e0f42b83a823baf0d9856d4347690791c"} Oct 30 13:16:48.708947 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:48.707677 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" event={"ID":"8493b4e1-efad-4d0f-bfef-28a37facefdb","Type":"ContainerStarted","Data":"6955984180a85db807d7ae87a7eb2adec4aa7e38fbd94ba01aeb74fd3932ec47"} Oct 30 13:16:48.708947 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:48.708883 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:16:48.785163 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:48.785103 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" podStartSLOduration=1.457936181 podStartE2EDuration="1.785088139s" totalImagesPullingTime="327.151958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:48.781400624 +0000 UTC m=+1465.413587787" watchObservedRunningTime="2026-10-30 13:16:48.785088139 +0000 UTC m=+1465.417275301" Oct 30 13:16:49.076820 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:49.076727 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57"] Oct 30 13:16:49.077317 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:49.076985 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" podUID="94ab8970-c142-48e4-9e5e-afbe6aa9f812" containerName="istio-proxy" containerID="cri-o://15fb9fa8a4716d55dcceade979af6f3912d220f3b0a794a876e63a6ca844cfbe" gracePeriod=30 Oct 30 13:16:52.445027 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:52.444987 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l"] Oct 30 13:16:52.449457 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:52.449436 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:16:52.488848 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:52.488812 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l"] Oct 30 13:16:52.530649 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:52.530608 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4644a1-serving-cert\"" Oct 30 13:16:52.603292 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:52.603259 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fe92cf8-8670-47c1-bff4-604674fd22cb-maas-api-tls\") pod \"maas-api-e2e-api-b-4644a1-74d9f89479-27l8l\" (UID: \"1fe92cf8-8670-47c1-bff4-604674fd22cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:16:52.603450 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:52.603314 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe92cf8-8670-47c1-bff4-604674fd22cb-openshift-service-ca\") pod \"maas-api-e2e-api-b-4644a1-74d9f89479-27l8l\" (UID: \"1fe92cf8-8670-47c1-bff4-604674fd22cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:16:52.603627 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:52.603595 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fe92cf8-8670-47c1-bff4-604674fd22cb-metrics-certs\") pod \"maas-api-e2e-api-b-4644a1-74d9f89479-27l8l\" (UID: \"1fe92cf8-8670-47c1-bff4-604674fd22cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:16:52.603770 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:52.603637 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx5kv\" (UniqueName: \"kubernetes.io/projected/1fe92cf8-8670-47c1-bff4-604674fd22cb-kube-api-access-qx5kv\") pod \"maas-api-e2e-api-b-4644a1-74d9f89479-27l8l\" (UID: \"1fe92cf8-8670-47c1-bff4-604674fd22cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:16:52.705684 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:52.705588 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fe92cf8-8670-47c1-bff4-604674fd22cb-metrics-certs\") pod \"maas-api-e2e-api-b-4644a1-74d9f89479-27l8l\" (UID: \"1fe92cf8-8670-47c1-bff4-604674fd22cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:16:52.705684 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:52.705653 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx5kv\" (UniqueName: \"kubernetes.io/projected/1fe92cf8-8670-47c1-bff4-604674fd22cb-kube-api-access-qx5kv\") pod \"maas-api-e2e-api-b-4644a1-74d9f89479-27l8l\" (UID: \"1fe92cf8-8670-47c1-bff4-604674fd22cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:16:52.705923 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:52.705795 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fe92cf8-8670-47c1-bff4-604674fd22cb-maas-api-tls\") pod \"maas-api-e2e-api-b-4644a1-74d9f89479-27l8l\" (UID: \"1fe92cf8-8670-47c1-bff4-604674fd22cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:16:52.705923 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:52.705838 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe92cf8-8670-47c1-bff4-604674fd22cb-openshift-service-ca\") pod \"maas-api-e2e-api-b-4644a1-74d9f89479-27l8l\" (UID: \"1fe92cf8-8670-47c1-bff4-604674fd22cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:16:52.706809 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:52.706787 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe92cf8-8670-47c1-bff4-604674fd22cb-openshift-service-ca\") pod \"maas-api-e2e-api-b-4644a1-74d9f89479-27l8l\" (UID: \"1fe92cf8-8670-47c1-bff4-604674fd22cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:16:52.707284 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:52.707257 7010 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-serving-cert: secret "maas-api-e2e-api-b-4644a1-serving-cert" not found Oct 30 13:16:52.707441 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:52.707352 7010 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe92cf8-8670-47c1-bff4-604674fd22cb-maas-api-tls podName:1fe92cf8-8670-47c1-bff4-604674fd22cb nodeName:}" failed. No retries permitted until 2026-10-30 13:16:53.207331138 +0000 UTC m=+1469.839518278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1fe92cf8-8670-47c1-bff4-604674fd22cb-maas-api-tls") pod "maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" (UID: "1fe92cf8-8670-47c1-bff4-604674fd22cb") : secret "maas-api-e2e-api-b-4644a1-serving-cert" not found Oct 30 13:16:52.709674 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:52.709650 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fe92cf8-8670-47c1-bff4-604674fd22cb-metrics-certs\") pod \"maas-api-e2e-api-b-4644a1-74d9f89479-27l8l\" (UID: \"1fe92cf8-8670-47c1-bff4-604674fd22cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:16:52.732838 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:52.732805 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx5kv\" (UniqueName: \"kubernetes.io/projected/1fe92cf8-8670-47c1-bff4-604674fd22cb-kube-api-access-qx5kv\") pod \"maas-api-e2e-api-b-4644a1-74d9f89479-27l8l\" (UID: \"1fe92cf8-8670-47c1-bff4-604674fd22cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:16:53.212799 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:53.212761 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fe92cf8-8670-47c1-bff4-604674fd22cb-maas-api-tls\") pod \"maas-api-e2e-api-b-4644a1-74d9f89479-27l8l\" (UID: \"1fe92cf8-8670-47c1-bff4-604674fd22cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:16:53.215323 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:53.215294 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fe92cf8-8670-47c1-bff4-604674fd22cb-maas-api-tls\") pod \"maas-api-e2e-api-b-4644a1-74d9f89479-27l8l\" (UID: \"1fe92cf8-8670-47c1-bff4-604674fd22cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:16:53.365176 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:53.365133 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:16:53.689572 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:16:53.689540 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe92cf8_8670_47c1_bff4_604674fd22cb.slice/crio-40914f60401283b33f280cec3662670e390230c95c35ab8f6ae9f7eff5b0fb85 WatchSource:0}: Error finding container 40914f60401283b33f280cec3662670e390230c95c35ab8f6ae9f7eff5b0fb85: Status 404 returned error can't find the container with id 40914f60401283b33f280cec3662670e390230c95c35ab8f6ae9f7eff5b0fb85 Oct 30 13:16:53.690027 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:53.689627 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l"] Oct 30 13:16:53.742720 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:53.742677 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" event={"ID":"1fe92cf8-8670-47c1-bff4-604674fd22cb","Type":"ContainerStarted","Data":"40914f60401283b33f280cec3662670e390230c95c35ab8f6ae9f7eff5b0fb85"} Oct 30 13:16:54.389280 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.389254 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:16:54.451407 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.451373 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-workload-socket\") pod \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " Oct 30 13:16:54.451407 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.451418 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-workload-certs\") pod \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " Oct 30 13:16:54.451685 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.451441 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-token\" (UniqueName: \"kubernetes.io/projected/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-token\") pod \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " Oct 30 13:16:54.451685 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.451465 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-data\") pod \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " Oct 30 13:16:54.451685 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.451521 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-podinfo\") pod \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " Oct 30 13:16:54.451685 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.451589 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istiod-ca-cert\") pod \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " Oct 30 13:16:54.451685 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.451607 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ab8970-c142-48e4-9e5e-afbe6aa9f812-kube-api-access-l75t9\" (UniqueName: \"kubernetes.io/projected/94ab8970-c142-48e4-9e5e-afbe6aa9f812-kube-api-access-l75t9\") pod \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " Oct 30 13:16:54.451685 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.451655 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-envoy\") pod \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " Oct 30 13:16:54.452019 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.451694 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-credential-socket\") pod \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\" (UID: \"94ab8970-c142-48e4-9e5e-afbe6aa9f812\") " Oct 30 13:16:54.452019 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.451775 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-workload-certs" pod "94ab8970-c142-48e4-9e5e-afbe6aa9f812" (UID: "94ab8970-c142-48e4-9e5e-afbe6aa9f812"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:54.452019 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.451794 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-workload-socket" pod "94ab8970-c142-48e4-9e5e-afbe6aa9f812" (UID: "94ab8970-c142-48e4-9e5e-afbe6aa9f812"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:54.452179 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.452046 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-data" pod "94ab8970-c142-48e4-9e5e-afbe6aa9f812" (UID: "94ab8970-c142-48e4-9e5e-afbe6aa9f812"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:54.452179 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.452106 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-credential-socket" pod "94ab8970-c142-48e4-9e5e-afbe6aa9f812" (UID: "94ab8970-c142-48e4-9e5e-afbe6aa9f812"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:54.452321 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.452294 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istiod-ca-cert" pod "94ab8970-c142-48e4-9e5e-afbe6aa9f812" (UID: "94ab8970-c142-48e4-9e5e-afbe6aa9f812"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:16:54.452570 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.452552 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.452640 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.452571 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.452640 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.452582 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.452640 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.452590 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.452640 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.452603 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.454430 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.454404 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-envoy" pod "94ab8970-c142-48e4-9e5e-afbe6aa9f812" (UID: "94ab8970-c142-48e4-9e5e-afbe6aa9f812"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:16:54.454820 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.454445 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-token" pod "94ab8970-c142-48e4-9e5e-afbe6aa9f812" (UID: "94ab8970-c142-48e4-9e5e-afbe6aa9f812"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:54.454820 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.454464 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-podinfo" pod "94ab8970-c142-48e4-9e5e-afbe6aa9f812" (UID: "94ab8970-c142-48e4-9e5e-afbe6aa9f812"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:16:54.454820 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.454471 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ab8970-c142-48e4-9e5e-afbe6aa9f812-kube-api-access-l75t9" pod "94ab8970-c142-48e4-9e5e-afbe6aa9f812" (UID: "94ab8970-c142-48e4-9e5e-afbe6aa9f812"). InnerVolumeSpecName "kube-api-access-l75t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:16:54.554142 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.554099 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.554142 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.554141 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.554386 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.554156 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l75t9\" (UniqueName: \"kubernetes.io/projected/94ab8970-c142-48e4-9e5e-afbe6aa9f812-kube-api-access-l75t9\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.554386 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.554169 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ab8970-c142-48e4-9e5e-afbe6aa9f812-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:16:54.753329 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.753293 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" event={"ID":"1fe92cf8-8670-47c1-bff4-604674fd22cb","Type":"ContainerStarted","Data":"24d47aac89ab06e4f038ef9fc2b50f34095f1835dd9a58bc970c89ab07e135e2"} Oct 30 13:16:54.753793 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.753514 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:16:54.755244 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.755216 7010 generic.go:356] "Generic (PLEG): container finished" podID="94ab8970-c142-48e4-9e5e-afbe6aa9f812" containerID="15fb9fa8a4716d55dcceade979af6f3912d220f3b0a794a876e63a6ca844cfbe" exitCode=0 Oct 30 13:16:54.755387 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.755272 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" Oct 30 13:16:54.755387 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.755304 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" event={"ID":"94ab8970-c142-48e4-9e5e-afbe6aa9f812","Type":"ContainerDied","Data":"15fb9fa8a4716d55dcceade979af6f3912d220f3b0a794a876e63a6ca844cfbe"} Oct 30 13:16:54.755387 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.755356 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57" event={"ID":"94ab8970-c142-48e4-9e5e-afbe6aa9f812","Type":"ContainerDied","Data":"e2ca04b4f1189568c9ce79fd58997521c5e160bd92b5729407c65b76d838f0bc"} Oct 30 13:16:54.755387 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.755386 7010 scope.go:122] "RemoveContainer" containerID="15fb9fa8a4716d55dcceade979af6f3912d220f3b0a794a876e63a6ca844cfbe" Oct 30 13:16:54.767822 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.767797 7010 scope.go:122] "RemoveContainer" containerID="12380aab9b447818abc37eb063a71268038643f79d2972e3eb5b12e41841ccb7" Oct 30 13:16:54.778289 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.778265 7010 scope.go:122] "RemoveContainer" containerID="15fb9fa8a4716d55dcceade979af6f3912d220f3b0a794a876e63a6ca844cfbe" Oct 30 13:16:54.778867 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:54.778839 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15fb9fa8a4716d55dcceade979af6f3912d220f3b0a794a876e63a6ca844cfbe\": container with ID starting with 15fb9fa8a4716d55dcceade979af6f3912d220f3b0a794a876e63a6ca844cfbe not found: ID does not exist" containerID="15fb9fa8a4716d55dcceade979af6f3912d220f3b0a794a876e63a6ca844cfbe" Oct 30 13:16:54.778970 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.778879 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15fb9fa8a4716d55dcceade979af6f3912d220f3b0a794a876e63a6ca844cfbe"} err="failed to get container status \"15fb9fa8a4716d55dcceade979af6f3912d220f3b0a794a876e63a6ca844cfbe\": rpc error: code = NotFound desc = could not find container \"15fb9fa8a4716d55dcceade979af6f3912d220f3b0a794a876e63a6ca844cfbe\": container with ID starting with 15fb9fa8a4716d55dcceade979af6f3912d220f3b0a794a876e63a6ca844cfbe not found: ID does not exist" Oct 30 13:16:54.778970 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.778905 7010 scope.go:122] "RemoveContainer" containerID="12380aab9b447818abc37eb063a71268038643f79d2972e3eb5b12e41841ccb7" Oct 30 13:16:54.779247 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:54.779227 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12380aab9b447818abc37eb063a71268038643f79d2972e3eb5b12e41841ccb7\": container with ID starting with 12380aab9b447818abc37eb063a71268038643f79d2972e3eb5b12e41841ccb7 not found: ID does not exist" containerID="12380aab9b447818abc37eb063a71268038643f79d2972e3eb5b12e41841ccb7" Oct 30 13:16:54.779349 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.779249 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12380aab9b447818abc37eb063a71268038643f79d2972e3eb5b12e41841ccb7"} err="failed to get container status \"12380aab9b447818abc37eb063a71268038643f79d2972e3eb5b12e41841ccb7\": rpc error: code = NotFound desc = could not find container \"12380aab9b447818abc37eb063a71268038643f79d2972e3eb5b12e41841ccb7\": container with ID starting with 12380aab9b447818abc37eb063a71268038643f79d2972e3eb5b12e41841ccb7 not found: ID does not exist" Oct 30 13:16:54.814793 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.814737 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" podStartSLOduration=2.368716883 podStartE2EDuration="2.814721247s" totalImagesPullingTime="446.004364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:54.813695246 +0000 UTC m=+1471.445882418" watchObservedRunningTime="2026-10-30 13:16:54.814721247 +0000 UTC m=+1471.446908409" Oct 30 13:16:54.870660 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.870627 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57"] Oct 30 13:16:54.878362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:54.878329 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fafb3f-openshift-default-5787656c65-bbp57"] Oct 30 13:16:55.490765 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:55.490732 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94ab8970-c142-48e4-9e5e-afbe6aa9f812" path="/var/lib/kubelet/pods/94ab8970-c142-48e4-9e5e-afbe6aa9f812/volumes" Oct 30 13:16:55.726161 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:16:55.726130 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:16:59.424352 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:16:59.424301 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:17:00.770678 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:00.770648 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:17:15.600674 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:15.600631 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:17:15.601187 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:15.600706 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" podUID="4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:17:17.964829 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:17.964792 7010 generic.go:356] "Generic (PLEG): container finished" podID="4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" containerID="d7ae7821917dc2438911ded9942372ca66f915b8ef34434bf5aafac3c9d0a874" exitCode=0 Oct 30 13:17:17.965390 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:17.964869 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" event={"ID":"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2","Type":"ContainerDied","Data":"d7ae7821917dc2438911ded9942372ca66f915b8ef34434bf5aafac3c9d0a874"} Oct 30 13:17:17.965481 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:17.965462 7010 scope.go:122] "RemoveContainer" containerID="d7ae7821917dc2438911ded9942372ca66f915b8ef34434bf5aafac3c9d0a874" Oct 30 13:17:17.967542 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:17.967502 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:17:17.967626 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:17.967572 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:17:17.967626 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:17.967606 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:17:18.975385 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:18.975348 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" event={"ID":"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2","Type":"ContainerStarted","Data":"8522aa41778bc42d1c0dcab71bd829765582c89ac4da7d3a836ba8ae71ba4408"} Oct 30 13:17:19.310915 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:19.310825 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:17:19.311076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:19.311044 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 13:17:19.311128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:19.311102 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" podUID="4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 13:17:20.311030 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:20.310879 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 13:17:20.311030 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:20.310944 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" podUID="4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 13:17:21.311381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:21.311341 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 13:17:21.311806 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:21.311402 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" podUID="4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 13:17:22.311080 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:22.311042 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 13:17:22.311265 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:22.311109 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" podUID="4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 13:17:23.310983 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:23.310949 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 13:17:23.311426 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:23.311004 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" podUID="4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 13:17:23.724465 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:23.724423 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:17:23.727525 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:23.727481 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:17:24.311341 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:24.311304 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 13:17:24.311791 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:24.311369 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" podUID="4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 13:17:25.315163 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:25.315130 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:17:25.315677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:25.315664 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:17:25.316335 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:25.316313 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:17:29.405165 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:17:29.405129 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:17:40.828760 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:40.828725 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8743-cd57f8"] Oct 30 13:17:40.829394 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:40.829367 7010 cpu_manager.go:414] "containerMap: removing container" podUID="94ab8970-c142-48e4-9e5e-afbe6aa9f812" containerName="istio-proxy" Oct 30 13:17:40.829451 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:40.829395 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ab8970-c142-48e4-9e5e-afbe6aa9f812" containerName="istio-proxy" Oct 30 13:17:40.829563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:40.829553 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ab8970-c142-48e4-9e5e-afbe6aa9f812" containerName="istio-proxy" Oct 30 13:17:40.829597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:40.829569 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ab8970-c142-48e4-9e5e-afbe6aa9f812" containerName="istio-proxy" Oct 30 13:17:40.833278 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:40.833257 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8743-cd57f8" Oct 30 13:17:40.851545 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:40.851463 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2jwn\" (UniqueName: \"kubernetes.io/projected/e8a31c9b-c486-4fe9-8034-789a843d0705-kube-api-access-s2jwn\") pod \"test-curl-8743-cd57f8\" (UID: \"e8a31c9b-c486-4fe9-8034-789a843d0705\") " pod="opendatahub/test-curl-8743-cd57f8" Oct 30 13:17:40.854529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:40.854484 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8743-cd57f8"] Oct 30 13:17:40.862978 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:40.862346 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nqkjx\"" Oct 30 13:17:40.952815 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:40.952774 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2jwn\" (UniqueName: \"kubernetes.io/projected/e8a31c9b-c486-4fe9-8034-789a843d0705-kube-api-access-s2jwn\") pod \"test-curl-8743-cd57f8\" (UID: \"e8a31c9b-c486-4fe9-8034-789a843d0705\") " pod="opendatahub/test-curl-8743-cd57f8" Oct 30 13:17:40.979362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:40.979329 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2jwn\" (UniqueName: \"kubernetes.io/projected/e8a31c9b-c486-4fe9-8034-789a843d0705-kube-api-access-s2jwn\") pod \"test-curl-8743-cd57f8\" (UID: \"e8a31c9b-c486-4fe9-8034-789a843d0705\") " pod="opendatahub/test-curl-8743-cd57f8" Oct 30 13:17:41.145834 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:41.145795 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8743-cd57f8" Oct 30 13:17:41.461230 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:41.461194 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8743-cd57f8"] Oct 30 13:17:41.463243 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:17:41.463214 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8a31c9b_c486_4fe9_8034_789a843d0705.slice/crio-e15ce2cdf3bddba58a934668bfe1f6a0cd2c3ff570ce10a572af3d4eb332f9ff WatchSource:0}: Error finding container e15ce2cdf3bddba58a934668bfe1f6a0cd2c3ff570ce10a572af3d4eb332f9ff: Status 404 returned error can't find the container with id e15ce2cdf3bddba58a934668bfe1f6a0cd2c3ff570ce10a572af3d4eb332f9ff Oct 30 13:17:42.207949 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:42.207909 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8743-cd57f8" event={"ID":"e8a31c9b-c486-4fe9-8034-789a843d0705","Type":"ContainerStarted","Data":"4953b9dfb62072a55103a2d69c877f9b7df1ff52702104a7dc6d603eff6c3482"} Oct 30 13:17:42.207949 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:42.207954 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8743-cd57f8" event={"ID":"e8a31c9b-c486-4fe9-8034-789a843d0705","Type":"ContainerStarted","Data":"e15ce2cdf3bddba58a934668bfe1f6a0cd2c3ff570ce10a572af3d4eb332f9ff"} Oct 30 13:17:42.248529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:42.248455 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8743-cd57f8" podStartSLOduration=2.248441113 podStartE2EDuration="2.248441113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:42.247719503 +0000 UTC m=+1518.879906665" watchObservedRunningTime="2026-10-30 13:17:42.248441113 +0000 UTC m=+1518.880628274" Oct 30 13:17:43.144111 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:43.144078 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8743-cd57f8"] Oct 30 13:17:43.151702 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:43.151664 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8743-cd57f8"] Oct 30 13:17:44.272332 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:44.272263 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8743-cd57f8" podUID="e8a31c9b-c486-4fe9-8034-789a843d0705" containerName="test-curl-8743-cd57f8" containerID="cri-o://4953b9dfb62072a55103a2d69c877f9b7df1ff52702104a7dc6d603eff6c3482" gracePeriod=2 Oct 30 13:17:46.502449 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:46.502421 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8743-cd57f8" Oct 30 13:17:46.566306 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:46.566224 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a31c9b-c486-4fe9-8034-789a843d0705-kube-api-access-s2jwn\" (UniqueName: \"kubernetes.io/projected/e8a31c9b-c486-4fe9-8034-789a843d0705-kube-api-access-s2jwn\") pod \"e8a31c9b-c486-4fe9-8034-789a843d0705\" (UID: \"e8a31c9b-c486-4fe9-8034-789a843d0705\") " Oct 30 13:17:46.568345 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:46.568317 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a31c9b-c486-4fe9-8034-789a843d0705-kube-api-access-s2jwn" pod "e8a31c9b-c486-4fe9-8034-789a843d0705" (UID: "e8a31c9b-c486-4fe9-8034-789a843d0705"). InnerVolumeSpecName "kube-api-access-s2jwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:46.668732 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:46.668696 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2jwn\" (UniqueName: \"kubernetes.io/projected/e8a31c9b-c486-4fe9-8034-789a843d0705-kube-api-access-s2jwn\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:17:47.300167 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:47.300127 7010 generic.go:356] "Generic (PLEG): container finished" podID="e8a31c9b-c486-4fe9-8034-789a843d0705" containerID="4953b9dfb62072a55103a2d69c877f9b7df1ff52702104a7dc6d603eff6c3482" exitCode=137 Oct 30 13:17:47.300392 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:47.300181 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8743-cd57f8" Oct 30 13:17:47.300392 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:47.300224 7010 scope.go:122] "RemoveContainer" containerID="4953b9dfb62072a55103a2d69c877f9b7df1ff52702104a7dc6d603eff6c3482" Oct 30 13:17:47.317186 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:47.317166 7010 scope.go:122] "RemoveContainer" containerID="4953b9dfb62072a55103a2d69c877f9b7df1ff52702104a7dc6d603eff6c3482" Oct 30 13:17:47.317568 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:17:47.317540 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4953b9dfb62072a55103a2d69c877f9b7df1ff52702104a7dc6d603eff6c3482\": container with ID starting with 4953b9dfb62072a55103a2d69c877f9b7df1ff52702104a7dc6d603eff6c3482 not found: ID does not exist" containerID="4953b9dfb62072a55103a2d69c877f9b7df1ff52702104a7dc6d603eff6c3482" Oct 30 13:17:47.317690 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:47.317574 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4953b9dfb62072a55103a2d69c877f9b7df1ff52702104a7dc6d603eff6c3482"} err="failed to get container status \"4953b9dfb62072a55103a2d69c877f9b7df1ff52702104a7dc6d603eff6c3482\": rpc error: code = NotFound desc = could not find container \"4953b9dfb62072a55103a2d69c877f9b7df1ff52702104a7dc6d603eff6c3482\": container with ID starting with 4953b9dfb62072a55103a2d69c877f9b7df1ff52702104a7dc6d603eff6c3482 not found: ID does not exist" Oct 30 13:17:47.327945 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:17:47.327909 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8743-cd57f8\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="e8a31c9b-c486-4fe9-8034-789a843d0705" pod="opendatahub/test-curl-8743-cd57f8" Oct 30 13:17:47.507526 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:47.500238 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a31c9b-c486-4fe9-8034-789a843d0705" path="/var/lib/kubelet/pods/e8a31c9b-c486-4fe9-8034-789a843d0705/volumes" Oct 30 13:17:50.010611 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:50.010570 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk"] Oct 30 13:17:55.012227 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.012123 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" podUID="58ed4375-0acf-48eb-97d3-2ccf5d1681f4" containerName="payload-processing" containerID="cri-o://3dcc1795a4fa0e42eef3efba8d715efe88803a32f5ae51433a4e302d9d0a0e5f" gracePeriod=25 Oct 30 13:17:55.141456 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:17:55.141429 7010 logging.go:55] [core] [Channel #77 SubChannel #78]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.84:50052", ServerName: "10.134.0.84:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.84:50052: connect: connection refused" Oct 30 13:17:55.297053 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.297027 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" Oct 30 13:17:55.428637 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.428603 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58ed4375-0acf-48eb-97d3-2ccf5d1681f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ed4375-0acf-48eb-97d3-2ccf5d1681f4-plugins-config-volume\") pod \"58ed4375-0acf-48eb-97d3-2ccf5d1681f4\" (UID: \"58ed4375-0acf-48eb-97d3-2ccf5d1681f4\") " Oct 30 13:17:55.428989 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.428966 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ed4375-0acf-48eb-97d3-2ccf5d1681f4-kube-api-access-76blk\" (UniqueName: \"kubernetes.io/projected/58ed4375-0acf-48eb-97d3-2ccf5d1681f4-kube-api-access-76blk\") pod \"58ed4375-0acf-48eb-97d3-2ccf5d1681f4\" (UID: \"58ed4375-0acf-48eb-97d3-2ccf5d1681f4\") " Oct 30 13:17:55.429100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.429038 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ed4375-0acf-48eb-97d3-2ccf5d1681f4-plugins-config-volume" pod "58ed4375-0acf-48eb-97d3-2ccf5d1681f4" (UID: "58ed4375-0acf-48eb-97d3-2ccf5d1681f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:17:55.429157 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.429114 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" Oct 30 13:17:55.429211 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.429160 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" event={"ID":"58ed4375-0acf-48eb-97d3-2ccf5d1681f4","Type":"ContainerDied","Data":"3dcc1795a4fa0e42eef3efba8d715efe88803a32f5ae51433a4e302d9d0a0e5f"} Oct 30 13:17:55.429264 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.429254 7010 scope.go:122] "RemoveContainer" containerID="3dcc1795a4fa0e42eef3efba8d715efe88803a32f5ae51433a4e302d9d0a0e5f" Oct 30 13:17:55.429603 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.429577 7010 generic.go:356] "Generic (PLEG): container finished" podID="58ed4375-0acf-48eb-97d3-2ccf5d1681f4" containerID="3dcc1795a4fa0e42eef3efba8d715efe88803a32f5ae51433a4e302d9d0a0e5f" exitCode=0 Oct 30 13:17:55.429891 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.429866 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ed4375-0acf-48eb-97d3-2ccf5d1681f4-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:17:55.429891 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.429881 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" event={"ID":"58ed4375-0acf-48eb-97d3-2ccf5d1681f4","Type":"ContainerDied","Data":"f73042d432c7a50ae640e124d9c02c74c97d24da76f2de69b378370d01ddff72"} Oct 30 13:17:55.431649 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.431575 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ed4375-0acf-48eb-97d3-2ccf5d1681f4-kube-api-access-76blk" pod "58ed4375-0acf-48eb-97d3-2ccf5d1681f4" (UID: "58ed4375-0acf-48eb-97d3-2ccf5d1681f4"). InnerVolumeSpecName "kube-api-access-76blk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:17:55.439500 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.439466 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn"] Oct 30 13:17:55.439689 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.439668 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" podUID="3891d8b5-1466-46b3-a720-de605147a7ba" containerName="istio-proxy" containerID="cri-o://12b19b00f36697d838e7394c10c26c0701660f82558cb806600dd40d6cf27ddc" gracePeriod=30 Oct 30 13:17:55.454720 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.454694 7010 scope.go:122] "RemoveContainer" containerID="3dcc1795a4fa0e42eef3efba8d715efe88803a32f5ae51433a4e302d9d0a0e5f" Oct 30 13:17:55.455147 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:17:55.455123 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dcc1795a4fa0e42eef3efba8d715efe88803a32f5ae51433a4e302d9d0a0e5f\": container with ID starting with 3dcc1795a4fa0e42eef3efba8d715efe88803a32f5ae51433a4e302d9d0a0e5f not found: ID does not exist" containerID="3dcc1795a4fa0e42eef3efba8d715efe88803a32f5ae51433a4e302d9d0a0e5f" Oct 30 13:17:55.455234 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.455158 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dcc1795a4fa0e42eef3efba8d715efe88803a32f5ae51433a4e302d9d0a0e5f"} err="failed to get container status \"3dcc1795a4fa0e42eef3efba8d715efe88803a32f5ae51433a4e302d9d0a0e5f\": rpc error: code = NotFound desc = could not find container \"3dcc1795a4fa0e42eef3efba8d715efe88803a32f5ae51433a4e302d9d0a0e5f\": container with ID starting with 3dcc1795a4fa0e42eef3efba8d715efe88803a32f5ae51433a4e302d9d0a0e5f not found: ID does not exist" Oct 30 13:17:55.531319 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.531226 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76blk\" (UniqueName: \"kubernetes.io/projected/58ed4375-0acf-48eb-97d3-2ccf5d1681f4-kube-api-access-76blk\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:17:55.770230 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.770194 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk"] Oct 30 13:17:55.774036 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:55.774013 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk"] Oct 30 13:17:56.141815 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:56.141764 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-8af0fe-678c6d76b-fphlk" podUID="58ed4375-0acf-48eb-97d3-2ccf5d1681f4" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.84:50052\" within 1s: context deadline exceeded" Oct 30 13:17:57.491602 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:57.491568 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58ed4375-0acf-48eb-97d3-2ccf5d1681f4" path="/var/lib/kubelet/pods/58ed4375-0acf-48eb-97d3-2ccf5d1681f4/volumes" Oct 30 13:17:59.416751 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:17:59.416709 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:17:59.540940 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:59.540904 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45"] Oct 30 13:17:59.541174 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:17:59.541132 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" podUID="0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" containerName="istio-proxy" containerID="cri-o://157cff6432a23db88530dd65157cf34b85d685cee2310982c216560a2e95a1d0" gracePeriod=30 Oct 30 13:18:00.730612 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.730586 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:18:00.750224 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.750191 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-credential-socket\") pod \"3891d8b5-1466-46b3-a720-de605147a7ba\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " Oct 30 13:18:00.750381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.750290 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-workload-certs\") pod \"3891d8b5-1466-46b3-a720-de605147a7ba\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " Oct 30 13:18:00.750381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.750341 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-istio-data\") pod \"3891d8b5-1466-46b3-a720-de605147a7ba\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " Oct 30 13:18:00.750381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.750374 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-istio-envoy\") pod \"3891d8b5-1466-46b3-a720-de605147a7ba\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " Oct 30 13:18:00.750582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.750395 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-workload-socket\") pod \"3891d8b5-1466-46b3-a720-de605147a7ba\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " Oct 30 13:18:00.750582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.750418 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3891d8b5-1466-46b3-a720-de605147a7ba-istio-token\" (UniqueName: \"kubernetes.io/projected/3891d8b5-1466-46b3-a720-de605147a7ba-istio-token\") pod \"3891d8b5-1466-46b3-a720-de605147a7ba\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " Oct 30 13:18:00.750582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.750448 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-credential-socket" pod "3891d8b5-1466-46b3-a720-de605147a7ba" (UID: "3891d8b5-1466-46b3-a720-de605147a7ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.750582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.750454 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3891d8b5-1466-46b3-a720-de605147a7ba-kube-api-access-s687m\" (UniqueName: \"kubernetes.io/projected/3891d8b5-1466-46b3-a720-de605147a7ba-kube-api-access-s687m\") pod \"3891d8b5-1466-46b3-a720-de605147a7ba\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " Oct 30 13:18:00.750790 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.750599 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3891d8b5-1466-46b3-a720-de605147a7ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3891d8b5-1466-46b3-a720-de605147a7ba-istio-podinfo\") pod \"3891d8b5-1466-46b3-a720-de605147a7ba\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " Oct 30 13:18:00.750790 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.750603 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-workload-certs" pod "3891d8b5-1466-46b3-a720-de605147a7ba" (UID: "3891d8b5-1466-46b3-a720-de605147a7ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.750790 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.750672 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3891d8b5-1466-46b3-a720-de605147a7ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3891d8b5-1466-46b3-a720-de605147a7ba-istiod-ca-cert\") pod \"3891d8b5-1466-46b3-a720-de605147a7ba\" (UID: \"3891d8b5-1466-46b3-a720-de605147a7ba\") " Oct 30 13:18:00.750790 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.750700 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-istio-data" pod "3891d8b5-1466-46b3-a720-de605147a7ba" (UID: "3891d8b5-1466-46b3-a720-de605147a7ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.750790 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.750776 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-workload-socket" pod "3891d8b5-1466-46b3-a720-de605147a7ba" (UID: "3891d8b5-1466-46b3-a720-de605147a7ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.751111 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.751091 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3891d8b5-1466-46b3-a720-de605147a7ba-istiod-ca-cert" pod "3891d8b5-1466-46b3-a720-de605147a7ba" (UID: "3891d8b5-1466-46b3-a720-de605147a7ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:00.751472 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.751447 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.751593 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.751477 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.751593 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.751508 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.751593 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.751522 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3891d8b5-1466-46b3-a720-de605147a7ba-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.751593 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.751537 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.752940 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.752913 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-istio-envoy" pod "3891d8b5-1466-46b3-a720-de605147a7ba" (UID: "3891d8b5-1466-46b3-a720-de605147a7ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:00.753038 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.752943 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3891d8b5-1466-46b3-a720-de605147a7ba-kube-api-access-s687m" pod "3891d8b5-1466-46b3-a720-de605147a7ba" (UID: "3891d8b5-1466-46b3-a720-de605147a7ba"). InnerVolumeSpecName "kube-api-access-s687m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:00.753207 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.753191 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3891d8b5-1466-46b3-a720-de605147a7ba-istio-token" pod "3891d8b5-1466-46b3-a720-de605147a7ba" (UID: "3891d8b5-1466-46b3-a720-de605147a7ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:00.753345 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.753329 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3891d8b5-1466-46b3-a720-de605147a7ba-istio-podinfo" pod "3891d8b5-1466-46b3-a720-de605147a7ba" (UID: "3891d8b5-1466-46b3-a720-de605147a7ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:18:00.852596 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.852558 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3891d8b5-1466-46b3-a720-de605147a7ba-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.852596 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.852588 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3891d8b5-1466-46b3-a720-de605147a7ba-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.852596 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.852598 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s687m\" (UniqueName: \"kubernetes.io/projected/3891d8b5-1466-46b3-a720-de605147a7ba-kube-api-access-s687m\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:00.852829 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:00.852606 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3891d8b5-1466-46b3-a720-de605147a7ba-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:01.479937 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:01.479902 7010 generic.go:356] "Generic (PLEG): container finished" podID="3891d8b5-1466-46b3-a720-de605147a7ba" containerID="12b19b00f36697d838e7394c10c26c0701660f82558cb806600dd40d6cf27ddc" exitCode=0 Oct 30 13:18:01.480116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:01.479949 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" Oct 30 13:18:01.480116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:01.479967 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" event={"ID":"3891d8b5-1466-46b3-a720-de605147a7ba","Type":"ContainerDied","Data":"12b19b00f36697d838e7394c10c26c0701660f82558cb806600dd40d6cf27ddc"} Oct 30 13:18:01.480116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:01.480010 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn" event={"ID":"3891d8b5-1466-46b3-a720-de605147a7ba","Type":"ContainerDied","Data":"64bedbcef8b6bfe65a4609bfe89cb915c8ab8ecbecb3c7482d58228b2bfef0c5"} Oct 30 13:18:01.480116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:01.480031 7010 scope.go:122] "RemoveContainer" containerID="12b19b00f36697d838e7394c10c26c0701660f82558cb806600dd40d6cf27ddc" Oct 30 13:18:01.491018 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:01.490990 7010 scope.go:122] "RemoveContainer" containerID="12b19b00f36697d838e7394c10c26c0701660f82558cb806600dd40d6cf27ddc" Oct 30 13:18:01.491328 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:18:01.491309 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b19b00f36697d838e7394c10c26c0701660f82558cb806600dd40d6cf27ddc\": container with ID starting with 12b19b00f36697d838e7394c10c26c0701660f82558cb806600dd40d6cf27ddc not found: ID does not exist" containerID="12b19b00f36697d838e7394c10c26c0701660f82558cb806600dd40d6cf27ddc" Oct 30 13:18:01.491390 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:01.491336 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b19b00f36697d838e7394c10c26c0701660f82558cb806600dd40d6cf27ddc"} err="failed to get container status \"12b19b00f36697d838e7394c10c26c0701660f82558cb806600dd40d6cf27ddc\": rpc error: code = NotFound desc = could not find container \"12b19b00f36697d838e7394c10c26c0701660f82558cb806600dd40d6cf27ddc\": container with ID starting with 12b19b00f36697d838e7394c10c26c0701660f82558cb806600dd40d6cf27ddc not found: ID does not exist" Oct 30 13:18:01.547583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:01.547551 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn"] Oct 30 13:18:01.560180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:01.560148 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-9whtn"] Oct 30 13:18:03.490112 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:03.490079 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3891d8b5-1466-46b3-a720-de605147a7ba" path="/var/lib/kubelet/pods/3891d8b5-1466-46b3-a720-de605147a7ba/volumes" Oct 30 13:18:04.347368 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.347217 7010 patch_prober.go:29] interesting pod/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:18:04.347368 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.347348 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" podUID="0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:18:04.825017 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.824989 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:18:04.914376 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.914285 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-credential-socket\") pod \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " Oct 30 13:18:04.914376 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.914340 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istiod-ca-cert\") pod \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " Oct 30 13:18:04.914376 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.914369 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-kube-api-access-w6v6h\" (UniqueName: \"kubernetes.io/projected/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-kube-api-access-w6v6h\") pod \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " Oct 30 13:18:04.914697 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.914550 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-podinfo\") pod \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " Oct 30 13:18:04.914697 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.914597 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-credential-socket" pod "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" (UID: "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:04.914810 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.914734 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-data\") pod \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " Oct 30 13:18:04.914810 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.914779 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istiod-ca-cert" pod "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" (UID: "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:04.914919 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.914864 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-envoy\") pod \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " Oct 30 13:18:04.914919 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.914903 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-workload-socket\") pod \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " Oct 30 13:18:04.915016 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.914928 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-workload-certs\") pod \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " Oct 30 13:18:04.915016 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.914987 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-token\" (UniqueName: \"kubernetes.io/projected/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-token\") pod \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\" (UID: \"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb\") " Oct 30 13:18:04.915574 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.915211 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-workload-certs" pod "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" (UID: "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:04.915574 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.915310 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-data" pod "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" (UID: "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:04.915574 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.915314 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-workload-socket" pod "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" (UID: "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:04.915840 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.915803 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:04.915840 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.915827 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:04.915944 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.915841 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:04.915944 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.915855 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:04.915944 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.915869 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:04.916989 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.916967 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-podinfo" pod "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" (UID: "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:18:04.917193 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.917174 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-kube-api-access-w6v6h" pod "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" (UID: "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb"). InnerVolumeSpecName "kube-api-access-w6v6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:04.917465 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.917446 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-envoy" pod "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" (UID: "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:04.917569 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:04.917530 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-token" pod "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" (UID: "0628c83b-2550-4bd0-aa9f-e0a8c869f0bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:05.017925 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:05.017889 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:05.017925 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:05.017923 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:05.017925 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:05.017933 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6v6h\" (UniqueName: \"kubernetes.io/projected/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-kube-api-access-w6v6h\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:05.017925 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:05.017943 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:05.507476 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:05.507438 7010 generic.go:356] "Generic (PLEG): container finished" podID="0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" containerID="157cff6432a23db88530dd65157cf34b85d685cee2310982c216560a2e95a1d0" exitCode=0 Oct 30 13:18:05.507622 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:05.507502 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" Oct 30 13:18:05.507622 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:05.507524 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" event={"ID":"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb","Type":"ContainerDied","Data":"157cff6432a23db88530dd65157cf34b85d685cee2310982c216560a2e95a1d0"} Oct 30 13:18:05.507622 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:05.507580 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45" event={"ID":"0628c83b-2550-4bd0-aa9f-e0a8c869f0bb","Type":"ContainerDied","Data":"8b60770071465e04a424a28165b1691f6457fd72284fc28041e0bf42b78f5449"} Oct 30 13:18:05.507622 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:05.507602 7010 scope.go:122] "RemoveContainer" containerID="157cff6432a23db88530dd65157cf34b85d685cee2310982c216560a2e95a1d0" Oct 30 13:18:05.517933 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:05.517913 7010 scope.go:122] "RemoveContainer" containerID="2b2e23a440240fe691fedecbf6d296595183aa9701365b04891203b93b90f843" Oct 30 13:18:05.529360 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:05.529340 7010 scope.go:122] "RemoveContainer" containerID="157cff6432a23db88530dd65157cf34b85d685cee2310982c216560a2e95a1d0" Oct 30 13:18:05.529696 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:18:05.529674 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"157cff6432a23db88530dd65157cf34b85d685cee2310982c216560a2e95a1d0\": container with ID starting with 157cff6432a23db88530dd65157cf34b85d685cee2310982c216560a2e95a1d0 not found: ID does not exist" containerID="157cff6432a23db88530dd65157cf34b85d685cee2310982c216560a2e95a1d0" Oct 30 13:18:05.529779 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:05.529709 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157cff6432a23db88530dd65157cf34b85d685cee2310982c216560a2e95a1d0"} err="failed to get container status \"157cff6432a23db88530dd65157cf34b85d685cee2310982c216560a2e95a1d0\": rpc error: code = NotFound desc = could not find container \"157cff6432a23db88530dd65157cf34b85d685cee2310982c216560a2e95a1d0\": container with ID starting with 157cff6432a23db88530dd65157cf34b85d685cee2310982c216560a2e95a1d0 not found: ID does not exist" Oct 30 13:18:05.529779 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:05.529732 7010 scope.go:122] "RemoveContainer" containerID="2b2e23a440240fe691fedecbf6d296595183aa9701365b04891203b93b90f843" Oct 30 13:18:05.530048 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:18:05.530030 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2e23a440240fe691fedecbf6d296595183aa9701365b04891203b93b90f843\": container with ID starting with 2b2e23a440240fe691fedecbf6d296595183aa9701365b04891203b93b90f843 not found: ID does not exist" containerID="2b2e23a440240fe691fedecbf6d296595183aa9701365b04891203b93b90f843" Oct 30 13:18:05.530090 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:05.530055 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2e23a440240fe691fedecbf6d296595183aa9701365b04891203b93b90f843"} err="failed to get container status \"2b2e23a440240fe691fedecbf6d296595183aa9701365b04891203b93b90f843\": rpc error: code = NotFound desc = could not find container \"2b2e23a440240fe691fedecbf6d296595183aa9701365b04891203b93b90f843\": container with ID starting with 2b2e23a440240fe691fedecbf6d296595183aa9701365b04891203b93b90f843 not found: ID does not exist" Oct 30 13:18:05.567027 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:05.566979 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45"] Oct 30 13:18:05.582261 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:05.582224 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-659d72-openshift-default-75dfdc9dd7-gnw45"] Oct 30 13:18:07.496338 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:07.496301 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" path="/var/lib/kubelet/pods/0628c83b-2550-4bd0-aa9f-e0a8c869f0bb/volumes" Oct 30 13:18:10.965124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:10.965084 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824"] Oct 30 13:18:10.966003 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:10.965939 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" podUID="8493b4e1-efad-4d0f-bfef-28a37facefdb" containerName="maas-api" containerID="cri-o://1d1e9f021945c033b4fdaa3de1748b6e0f42b83a823baf0d9856d4347690791c" gracePeriod=30 Oct 30 13:18:11.457508 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.456954 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg"] Oct 30 13:18:11.457508 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.457208 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" podUID="1c26e7b5-7225-4121-aed4-332a5028b843" containerName="maas-api" containerID="cri-o://ed917a83dd7ce4ae1f2d7b166330d639f948620788b2b570ef0ceb6e5e1beb6d" gracePeriod=30 Oct 30 13:18:11.605541 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.596231 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:18:11.637919 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.637879 7010 generic.go:356] "Generic (PLEG): container finished" podID="8493b4e1-efad-4d0f-bfef-28a37facefdb" containerID="1d1e9f021945c033b4fdaa3de1748b6e0f42b83a823baf0d9856d4347690791c" exitCode=0 Oct 30 13:18:11.638608 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.638149 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" event={"ID":"8493b4e1-efad-4d0f-bfef-28a37facefdb","Type":"ContainerDied","Data":"1d1e9f021945c033b4fdaa3de1748b6e0f42b83a823baf0d9856d4347690791c"} Oct 30 13:18:11.639043 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.638881 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" event={"ID":"8493b4e1-efad-4d0f-bfef-28a37facefdb","Type":"ContainerDied","Data":"6955984180a85db807d7ae87a7eb2adec4aa7e38fbd94ba01aeb74fd3932ec47"} Oct 30 13:18:11.639186 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.638268 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824" Oct 30 13:18:11.639517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.639020 7010 scope.go:122] "RemoveContainer" containerID="1d1e9f021945c033b4fdaa3de1748b6e0f42b83a823baf0d9856d4347690791c" Oct 30 13:18:11.694370 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.694333 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8493b4e1-efad-4d0f-bfef-28a37facefdb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8493b4e1-efad-4d0f-bfef-28a37facefdb-metrics-certs\") pod \"8493b4e1-efad-4d0f-bfef-28a37facefdb\" (UID: \"8493b4e1-efad-4d0f-bfef-28a37facefdb\") " Oct 30 13:18:11.695673 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.695647 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8493b4e1-efad-4d0f-bfef-28a37facefdb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8493b4e1-efad-4d0f-bfef-28a37facefdb-openshift-service-ca\") pod \"8493b4e1-efad-4d0f-bfef-28a37facefdb\" (UID: \"8493b4e1-efad-4d0f-bfef-28a37facefdb\") " Oct 30 13:18:11.696353 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.696331 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8493b4e1-efad-4d0f-bfef-28a37facefdb-kube-api-access-wqr7h\" (UniqueName: \"kubernetes.io/projected/8493b4e1-efad-4d0f-bfef-28a37facefdb-kube-api-access-wqr7h\") pod \"8493b4e1-efad-4d0f-bfef-28a37facefdb\" (UID: \"8493b4e1-efad-4d0f-bfef-28a37facefdb\") " Oct 30 13:18:11.696950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.696763 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8493b4e1-efad-4d0f-bfef-28a37facefdb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8493b4e1-efad-4d0f-bfef-28a37facefdb-maas-api-tls\") pod \"8493b4e1-efad-4d0f-bfef-28a37facefdb\" (UID: \"8493b4e1-efad-4d0f-bfef-28a37facefdb\") " Oct 30 13:18:11.699406 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.699375 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8493b4e1-efad-4d0f-bfef-28a37facefdb-openshift-service-ca" pod "8493b4e1-efad-4d0f-bfef-28a37facefdb" (UID: "8493b4e1-efad-4d0f-bfef-28a37facefdb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:11.701266 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.700752 7010 scope.go:122] "RemoveContainer" containerID="1d1e9f021945c033b4fdaa3de1748b6e0f42b83a823baf0d9856d4347690791c" Oct 30 13:18:11.701375 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:18:11.701317 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d1e9f021945c033b4fdaa3de1748b6e0f42b83a823baf0d9856d4347690791c\": container with ID starting with 1d1e9f021945c033b4fdaa3de1748b6e0f42b83a823baf0d9856d4347690791c not found: ID does not exist" containerID="1d1e9f021945c033b4fdaa3de1748b6e0f42b83a823baf0d9856d4347690791c" Oct 30 13:18:11.701442 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.701353 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1e9f021945c033b4fdaa3de1748b6e0f42b83a823baf0d9856d4347690791c"} err="failed to get container status \"1d1e9f021945c033b4fdaa3de1748b6e0f42b83a823baf0d9856d4347690791c\": rpc error: code = NotFound desc = could not find container \"1d1e9f021945c033b4fdaa3de1748b6e0f42b83a823baf0d9856d4347690791c\": container with ID starting with 1d1e9f021945c033b4fdaa3de1748b6e0f42b83a823baf0d9856d4347690791c not found: ID does not exist" Oct 30 13:18:11.740354 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.735188 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8493b4e1-efad-4d0f-bfef-28a37facefdb-maas-api-tls" pod "8493b4e1-efad-4d0f-bfef-28a37facefdb" (UID: "8493b4e1-efad-4d0f-bfef-28a37facefdb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:18:11.765818 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.765752 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8493b4e1-efad-4d0f-bfef-28a37facefdb-kube-api-access-wqr7h" pod "8493b4e1-efad-4d0f-bfef-28a37facefdb" (UID: "8493b4e1-efad-4d0f-bfef-28a37facefdb"). InnerVolumeSpecName "kube-api-access-wqr7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:11.766143 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.766103 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8493b4e1-efad-4d0f-bfef-28a37facefdb-metrics-certs" pod "8493b4e1-efad-4d0f-bfef-28a37facefdb" (UID: "8493b4e1-efad-4d0f-bfef-28a37facefdb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:18:11.809886 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.805803 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqr7h\" (UniqueName: \"kubernetes.io/projected/8493b4e1-efad-4d0f-bfef-28a37facefdb-kube-api-access-wqr7h\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:11.809886 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.805849 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8493b4e1-efad-4d0f-bfef-28a37facefdb-maas-api-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:11.809886 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.805864 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8493b4e1-efad-4d0f-bfef-28a37facefdb-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:11.809886 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:11.805877 7010 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8493b4e1-efad-4d0f-bfef-28a37facefdb-openshift-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:12.042539 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.042448 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824"] Oct 30 13:18:12.079794 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.079752 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:18:12.091673 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.083597 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6c3661-6695779d4-hg824"] Oct 30 13:18:12.124301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.118314 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c26e7b5-7225-4121-aed4-332a5028b843-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c26e7b5-7225-4121-aed4-332a5028b843-openshift-service-ca\") pod \"1c26e7b5-7225-4121-aed4-332a5028b843\" (UID: \"1c26e7b5-7225-4121-aed4-332a5028b843\") " Oct 30 13:18:12.124301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.118389 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c26e7b5-7225-4121-aed4-332a5028b843-kube-api-access-5fgp6\" (UniqueName: \"kubernetes.io/projected/1c26e7b5-7225-4121-aed4-332a5028b843-kube-api-access-5fgp6\") pod \"1c26e7b5-7225-4121-aed4-332a5028b843\" (UID: \"1c26e7b5-7225-4121-aed4-332a5028b843\") " Oct 30 13:18:12.124301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.118441 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c26e7b5-7225-4121-aed4-332a5028b843-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c26e7b5-7225-4121-aed4-332a5028b843-maas-api-tls\") pod \"1c26e7b5-7225-4121-aed4-332a5028b843\" (UID: \"1c26e7b5-7225-4121-aed4-332a5028b843\") " Oct 30 13:18:12.124301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.118751 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c26e7b5-7225-4121-aed4-332a5028b843-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c26e7b5-7225-4121-aed4-332a5028b843-metrics-certs\") pod \"1c26e7b5-7225-4121-aed4-332a5028b843\" (UID: \"1c26e7b5-7225-4121-aed4-332a5028b843\") " Oct 30 13:18:12.124301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.120363 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c26e7b5-7225-4121-aed4-332a5028b843-openshift-service-ca" pod "1c26e7b5-7225-4121-aed4-332a5028b843" (UID: "1c26e7b5-7225-4121-aed4-332a5028b843"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:18:12.128061 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.125319 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c26e7b5-7225-4121-aed4-332a5028b843-kube-api-access-5fgp6" pod "1c26e7b5-7225-4121-aed4-332a5028b843" (UID: "1c26e7b5-7225-4121-aed4-332a5028b843"). InnerVolumeSpecName "kube-api-access-5fgp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:12.128061 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.125465 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c26e7b5-7225-4121-aed4-332a5028b843-metrics-certs" pod "1c26e7b5-7225-4121-aed4-332a5028b843" (UID: "1c26e7b5-7225-4121-aed4-332a5028b843"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:18:12.128061 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.128447 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c26e7b5-7225-4121-aed4-332a5028b843-maas-api-tls" pod "1c26e7b5-7225-4121-aed4-332a5028b843" (UID: "1c26e7b5-7225-4121-aed4-332a5028b843"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:18:12.220978 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.220885 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c26e7b5-7225-4121-aed4-332a5028b843-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:12.220978 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.220923 7010 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c26e7b5-7225-4121-aed4-332a5028b843-openshift-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:12.220978 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.220937 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fgp6\" (UniqueName: \"kubernetes.io/projected/1c26e7b5-7225-4121-aed4-332a5028b843-kube-api-access-5fgp6\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:12.220978 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.220950 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c26e7b5-7225-4121-aed4-332a5028b843-maas-api-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:12.648786 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.648750 7010 generic.go:356] "Generic (PLEG): container finished" podID="1c26e7b5-7225-4121-aed4-332a5028b843" containerID="ed917a83dd7ce4ae1f2d7b166330d639f948620788b2b570ef0ceb6e5e1beb6d" exitCode=0 Oct 30 13:18:12.648975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.648810 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" Oct 30 13:18:12.648975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.648832 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" event={"ID":"1c26e7b5-7225-4121-aed4-332a5028b843","Type":"ContainerDied","Data":"ed917a83dd7ce4ae1f2d7b166330d639f948620788b2b570ef0ceb6e5e1beb6d"} Oct 30 13:18:12.648975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.648901 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg" event={"ID":"1c26e7b5-7225-4121-aed4-332a5028b843","Type":"ContainerDied","Data":"f1f6d5da71e594b7593f186203286bdd7ca21ef4681c0a5a710629191b32df43"} Oct 30 13:18:12.648975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.648933 7010 scope.go:122] "RemoveContainer" containerID="ed917a83dd7ce4ae1f2d7b166330d639f948620788b2b570ef0ceb6e5e1beb6d" Oct 30 13:18:12.662185 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.662161 7010 scope.go:122] "RemoveContainer" containerID="ed917a83dd7ce4ae1f2d7b166330d639f948620788b2b570ef0ceb6e5e1beb6d" Oct 30 13:18:12.662589 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:18:12.662560 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed917a83dd7ce4ae1f2d7b166330d639f948620788b2b570ef0ceb6e5e1beb6d\": container with ID starting with ed917a83dd7ce4ae1f2d7b166330d639f948620788b2b570ef0ceb6e5e1beb6d not found: ID does not exist" containerID="ed917a83dd7ce4ae1f2d7b166330d639f948620788b2b570ef0ceb6e5e1beb6d" Oct 30 13:18:12.662697 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.662603 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed917a83dd7ce4ae1f2d7b166330d639f948620788b2b570ef0ceb6e5e1beb6d"} err="failed to get container status \"ed917a83dd7ce4ae1f2d7b166330d639f948620788b2b570ef0ceb6e5e1beb6d\": rpc error: code = NotFound desc = could not find container \"ed917a83dd7ce4ae1f2d7b166330d639f948620788b2b570ef0ceb6e5e1beb6d\": container with ID starting with ed917a83dd7ce4ae1f2d7b166330d639f948620788b2b570ef0ceb6e5e1beb6d not found: ID does not exist" Oct 30 13:18:12.720255 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.719850 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg"] Oct 30 13:18:12.736721 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:12.736647 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6b0881-7db7ff89cb-fc2bg"] Oct 30 13:18:13.492502 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:13.492459 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c26e7b5-7225-4121-aed4-332a5028b843" path="/var/lib/kubelet/pods/1c26e7b5-7225-4121-aed4-332a5028b843/volumes" Oct 30 13:18:13.492993 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:13.492977 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8493b4e1-efad-4d0f-bfef-28a37facefdb" path="/var/lib/kubelet/pods/8493b4e1-efad-4d0f-bfef-28a37facefdb/volumes" Oct 30 13:18:17.734181 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:17.734142 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf"] Oct 30 13:18:27.546116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.546078 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z"] Oct 30 13:18:27.546985 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.546951 7010 cpu_manager.go:414] "containerMap: removing container" podUID="3891d8b5-1466-46b3-a720-de605147a7ba" containerName="istio-proxy" Oct 30 13:18:27.546985 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.546977 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3891d8b5-1466-46b3-a720-de605147a7ba" containerName="istio-proxy" Oct 30 13:18:27.547139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.546996 7010 cpu_manager.go:414] "containerMap: removing container" podUID="0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" containerName="istio-proxy" Oct 30 13:18:27.547139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547003 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" containerName="istio-proxy" Oct 30 13:18:27.547139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547020 7010 cpu_manager.go:414] "containerMap: removing container" podUID="8493b4e1-efad-4d0f-bfef-28a37facefdb" containerName="maas-api" Oct 30 13:18:27.547139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547028 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8493b4e1-efad-4d0f-bfef-28a37facefdb" containerName="maas-api" Oct 30 13:18:27.547139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547047 7010 cpu_manager.go:414] "containerMap: removing container" podUID="1c26e7b5-7225-4121-aed4-332a5028b843" containerName="maas-api" Oct 30 13:18:27.547139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547054 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c26e7b5-7225-4121-aed4-332a5028b843" containerName="maas-api" Oct 30 13:18:27.547139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547071 7010 cpu_manager.go:414] "containerMap: removing container" podUID="e8a31c9b-c486-4fe9-8034-789a843d0705" containerName="test-curl-8743-cd57f8" Oct 30 13:18:27.547139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547078 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a31c9b-c486-4fe9-8034-789a843d0705" containerName="test-curl-8743-cd57f8" Oct 30 13:18:27.547139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547115 7010 cpu_manager.go:414] "containerMap: removing container" podUID="0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" containerName="istio-proxy" Oct 30 13:18:27.547139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547122 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" containerName="istio-proxy" Oct 30 13:18:27.547139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547134 7010 cpu_manager.go:414] "containerMap: removing container" podUID="58ed4375-0acf-48eb-97d3-2ccf5d1681f4" containerName="payload-processing" Oct 30 13:18:27.547139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547143 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ed4375-0acf-48eb-97d3-2ccf5d1681f4" containerName="payload-processing" Oct 30 13:18:27.547535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547162 7010 cpu_manager.go:414] "containerMap: removing container" podUID="94ab8970-c142-48e4-9e5e-afbe6aa9f812" containerName="istio-proxy" Oct 30 13:18:27.547535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547169 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ab8970-c142-48e4-9e5e-afbe6aa9f812" containerName="istio-proxy" Oct 30 13:18:27.547535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547329 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="3891d8b5-1466-46b3-a720-de605147a7ba" containerName="istio-proxy" Oct 30 13:18:27.547535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547343 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" containerName="istio-proxy" Oct 30 13:18:27.547535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547350 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="8493b4e1-efad-4d0f-bfef-28a37facefdb" containerName="maas-api" Oct 30 13:18:27.547535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547356 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c26e7b5-7225-4121-aed4-332a5028b843" containerName="maas-api" Oct 30 13:18:27.547535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547362 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a31c9b-c486-4fe9-8034-789a843d0705" containerName="test-curl-8743-cd57f8" Oct 30 13:18:27.547535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547371 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="0628c83b-2550-4bd0-aa9f-e0a8c869f0bb" containerName="istio-proxy" Oct 30 13:18:27.547535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.547378 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ed4375-0acf-48eb-97d3-2ccf5d1681f4" containerName="payload-processing" Oct 30 13:18:27.552471 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.552453 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" Oct 30 13:18:27.569579 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.569550 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z"] Oct 30 13:18:27.570778 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.570756 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b5a369\"" Oct 30 13:18:27.578904 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.578873 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b5a369-dockercfg-ww6fm\"" Oct 30 13:18:27.656514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.634923 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f572a662-3ef2-4230-a1f5-6bf0ae106344-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z\" (UID: \"f572a662-3ef2-4230-a1f5-6bf0ae106344\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" Oct 30 13:18:27.656514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.635021 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw8tq\" (UniqueName: \"kubernetes.io/projected/f572a662-3ef2-4230-a1f5-6bf0ae106344-kube-api-access-hw8tq\") pod \"payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z\" (UID: \"f572a662-3ef2-4230-a1f5-6bf0ae106344\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" Oct 30 13:18:27.738545 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.736465 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f572a662-3ef2-4230-a1f5-6bf0ae106344-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z\" (UID: \"f572a662-3ef2-4230-a1f5-6bf0ae106344\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" Oct 30 13:18:27.738545 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.736560 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw8tq\" (UniqueName: \"kubernetes.io/projected/f572a662-3ef2-4230-a1f5-6bf0ae106344-kube-api-access-hw8tq\") pod \"payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z\" (UID: \"f572a662-3ef2-4230-a1f5-6bf0ae106344\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" Oct 30 13:18:27.738545 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.737710 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f572a662-3ef2-4230-a1f5-6bf0ae106344-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z\" (UID: \"f572a662-3ef2-4230-a1f5-6bf0ae106344\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" Oct 30 13:18:27.790360 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.790318 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw8tq\" (UniqueName: \"kubernetes.io/projected/f572a662-3ef2-4230-a1f5-6bf0ae106344-kube-api-access-hw8tq\") pod \"payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z\" (UID: \"f572a662-3ef2-4230-a1f5-6bf0ae106344\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" Oct 30 13:18:27.866102 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:27.865985 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" Oct 30 13:18:28.180430 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:28.180395 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z"] Oct 30 13:18:28.182307 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:18:28.182275 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf572a662_3ef2_4230_a1f5_6bf0ae106344.slice/crio-6532c8b2c74fe4c0d333dcc0539a9430041af2bd776d80bb6665b583724b563e WatchSource:0}: Error finding container 6532c8b2c74fe4c0d333dcc0539a9430041af2bd776d80bb6665b583724b563e: Status 404 returned error can't find the container with id 6532c8b2c74fe4c0d333dcc0539a9430041af2bd776d80bb6665b583724b563e Oct 30 13:18:28.830631 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:28.830582 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" event={"ID":"f572a662-3ef2-4230-a1f5-6bf0ae106344","Type":"ContainerStarted","Data":"ac8582d7c6917ae1fcebc6059e0a100ceca867cdb88a9d6d32563ef0f6139bd0"} Oct 30 13:18:28.831023 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:28.830652 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" Oct 30 13:18:28.831023 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:28.830672 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" event={"ID":"f572a662-3ef2-4230-a1f5-6bf0ae106344","Type":"ContainerStarted","Data":"6532c8b2c74fe4c0d333dcc0539a9430041af2bd776d80bb6665b583724b563e"} Oct 30 13:18:28.879142 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:28.879091 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" podStartSLOduration=1.8790781650000001 podStartE2EDuration="1.879078165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:28.877874241 +0000 UTC m=+1565.510061405" watchObservedRunningTime="2026-10-30 13:18:28.879078165 +0000 UTC m=+1565.511265364" Oct 30 13:18:29.400827 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:18:29.400787 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:18:32.759819 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:32.759753 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" podUID="8042b023-65f6-4cc1-a2b6-5f94eead7207" containerName="main" containerID="cri-o://36f5ff33515a43ed760a12e3872f91c61b34c9f9a24e34a9e67b148f3af94a0a" gracePeriod=45 Oct 30 13:18:33.039354 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.039328 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:18:33.226934 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.226899 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8042b023-65f6-4cc1-a2b6-5f94eead7207-kube-api-access-lfbzl\" (UniqueName: \"kubernetes.io/projected/8042b023-65f6-4cc1-a2b6-5f94eead7207-kube-api-access-lfbzl\") pod \"8042b023-65f6-4cc1-a2b6-5f94eead7207\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " Oct 30 13:18:33.227141 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.227121 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-model-cache\") pod \"8042b023-65f6-4cc1-a2b6-5f94eead7207\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " Oct 30 13:18:33.227201 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.227173 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8042b023-65f6-4cc1-a2b6-5f94eead7207-tls-certs\" (UniqueName: \"kubernetes.io/secret/8042b023-65f6-4cc1-a2b6-5f94eead7207-tls-certs\") pod \"8042b023-65f6-4cc1-a2b6-5f94eead7207\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " Oct 30 13:18:33.227264 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.227230 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-home\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-home\") pod \"8042b023-65f6-4cc1-a2b6-5f94eead7207\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " Oct 30 13:18:33.227393 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.227368 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-tmp-dir\") pod \"8042b023-65f6-4cc1-a2b6-5f94eead7207\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " Oct 30 13:18:33.227558 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.227415 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-dshm\") pod \"8042b023-65f6-4cc1-a2b6-5f94eead7207\" (UID: \"8042b023-65f6-4cc1-a2b6-5f94eead7207\") " Oct 30 13:18:33.227558 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.227421 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-model-cache" pod "8042b023-65f6-4cc1-a2b6-5f94eead7207" (UID: "8042b023-65f6-4cc1-a2b6-5f94eead7207"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:33.227691 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.227557 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-home" pod "8042b023-65f6-4cc1-a2b6-5f94eead7207" (UID: "8042b023-65f6-4cc1-a2b6-5f94eead7207"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:33.227691 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.227651 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-tmp-dir" pod "8042b023-65f6-4cc1-a2b6-5f94eead7207" (UID: "8042b023-65f6-4cc1-a2b6-5f94eead7207"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:33.228096 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.228074 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:33.228150 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.228104 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:33.228150 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.228119 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:33.229567 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.229506 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-dshm" pod "8042b023-65f6-4cc1-a2b6-5f94eead7207" (UID: "8042b023-65f6-4cc1-a2b6-5f94eead7207"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:18:33.229567 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.229513 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8042b023-65f6-4cc1-a2b6-5f94eead7207-tls-certs" pod "8042b023-65f6-4cc1-a2b6-5f94eead7207" (UID: "8042b023-65f6-4cc1-a2b6-5f94eead7207"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:18:33.229718 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.229620 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8042b023-65f6-4cc1-a2b6-5f94eead7207-kube-api-access-lfbzl" pod "8042b023-65f6-4cc1-a2b6-5f94eead7207" (UID: "8042b023-65f6-4cc1-a2b6-5f94eead7207"). InnerVolumeSpecName "kube-api-access-lfbzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:18:33.329293 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.329222 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8042b023-65f6-4cc1-a2b6-5f94eead7207-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:33.329293 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.329256 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfbzl\" (UniqueName: \"kubernetes.io/projected/8042b023-65f6-4cc1-a2b6-5f94eead7207-kube-api-access-lfbzl\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:33.329293 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.329274 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8042b023-65f6-4cc1-a2b6-5f94eead7207-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:18:33.863760 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.863727 7010 generic.go:356] "Generic (PLEG): container finished" podID="8042b023-65f6-4cc1-a2b6-5f94eead7207" containerID="36f5ff33515a43ed760a12e3872f91c61b34c9f9a24e34a9e67b148f3af94a0a" exitCode=0 Oct 30 13:18:33.864308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.863770 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" event={"ID":"8042b023-65f6-4cc1-a2b6-5f94eead7207","Type":"ContainerDied","Data":"36f5ff33515a43ed760a12e3872f91c61b34c9f9a24e34a9e67b148f3af94a0a"} Oct 30 13:18:33.864308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.863787 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" Oct 30 13:18:33.864308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.863797 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" event={"ID":"8042b023-65f6-4cc1-a2b6-5f94eead7207","Type":"ContainerDied","Data":"78796a6a59cfc71492b4913e04184f36fe56a9d74f933c61e9d6141183d2bc55"} Oct 30 13:18:33.864308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.863812 7010 scope.go:122] "RemoveContainer" containerID="36f5ff33515a43ed760a12e3872f91c61b34c9f9a24e34a9e67b148f3af94a0a" Oct 30 13:18:33.874446 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.874425 7010 scope.go:122] "RemoveContainer" containerID="36f5ff33515a43ed760a12e3872f91c61b34c9f9a24e34a9e67b148f3af94a0a" Oct 30 13:18:33.874788 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:18:33.874769 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f5ff33515a43ed760a12e3872f91c61b34c9f9a24e34a9e67b148f3af94a0a\": container with ID starting with 36f5ff33515a43ed760a12e3872f91c61b34c9f9a24e34a9e67b148f3af94a0a not found: ID does not exist" containerID="36f5ff33515a43ed760a12e3872f91c61b34c9f9a24e34a9e67b148f3af94a0a" Oct 30 13:18:33.874868 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.874798 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f5ff33515a43ed760a12e3872f91c61b34c9f9a24e34a9e67b148f3af94a0a"} err="failed to get container status \"36f5ff33515a43ed760a12e3872f91c61b34c9f9a24e34a9e67b148f3af94a0a\": rpc error: code = NotFound desc = could not find container \"36f5ff33515a43ed760a12e3872f91c61b34c9f9a24e34a9e67b148f3af94a0a\": container with ID starting with 36f5ff33515a43ed760a12e3872f91c61b34c9f9a24e34a9e67b148f3af94a0a not found: ID does not exist" Oct 30 13:18:33.917435 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.917399 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf"] Oct 30 13:18:33.930087 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.930061 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf"] Oct 30 13:18:33.965547 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:33.965506 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-df6c3661/e2e-lifecycle-model-df6c3661-kserve-b88d69fbf-jbpgf" podUID="8042b023-65f6-4cc1-a2b6-5f94eead7207" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:18:35.490367 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:35.490334 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8042b023-65f6-4cc1-a2b6-5f94eead7207" path="/var/lib/kubelet/pods/8042b023-65f6-4cc1-a2b6-5f94eead7207/volumes" Oct 30 13:18:39.839383 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:39.839350 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" Oct 30 13:18:57.662878 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:57.662837 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb"] Oct 30 13:18:57.663571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:18:57.663092 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" podUID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" containerName="istio-proxy" containerID="cri-o://35aafcc6a759bb96be6d83de18da24367b5473bf0e623a0f02cfd7820d9bb0f4" gracePeriod=30 Oct 30 13:18:59.403894 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:18:59.403851 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:19:00.620137 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.620090 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd"] Oct 30 13:19:00.621075 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.621050 7010 cpu_manager.go:414] "containerMap: removing container" podUID="8042b023-65f6-4cc1-a2b6-5f94eead7207" containerName="main" Oct 30 13:19:00.621196 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.621077 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8042b023-65f6-4cc1-a2b6-5f94eead7207" containerName="main" Oct 30 13:19:00.621327 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.621305 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="8042b023-65f6-4cc1-a2b6-5f94eead7207" containerName="main" Oct 30 13:19:00.626960 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.626935 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:19:00.643057 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.643019 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd"] Oct 30 13:19:00.651948 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.651762 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b5a369-serving-cert\"" Oct 30 13:19:00.666519 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.666050 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-maas-api-tls\") pod \"maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd\" (UID: \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:19:00.666519 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.666119 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd\" (UID: \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:19:00.666519 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.666145 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j4rk\" (UniqueName: \"kubernetes.io/projected/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-kube-api-access-6j4rk\") pod \"maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd\" (UID: \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:19:00.666519 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.666209 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-metrics-certs\") pod \"maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd\" (UID: \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:19:00.767771 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.767734 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd\" (UID: \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:19:00.767771 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.767775 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j4rk\" (UniqueName: \"kubernetes.io/projected/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-kube-api-access-6j4rk\") pod \"maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd\" (UID: \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:19:00.768023 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.767867 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-metrics-certs\") pod \"maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd\" (UID: \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:19:00.768103 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.768083 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-maas-api-tls\") pod \"maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd\" (UID: \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:19:00.768298 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:19:00.768280 7010 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-serving-cert: secret "maas-api-e2e-inf-a-b5a369-serving-cert" not found Oct 30 13:19:00.768360 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:19:00.768349 7010 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-maas-api-tls podName:3f0de2c5-3253-402e-99ec-70d8fb93a8e8 nodeName:}" failed. No retries permitted until 2026-10-30 13:19:01.268330352 +0000 UTC m=+1597.900517505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-maas-api-tls") pod "maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" (UID: "3f0de2c5-3253-402e-99ec-70d8fb93a8e8") : secret "maas-api-e2e-inf-a-b5a369-serving-cert" not found Oct 30 13:19:00.768582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.768552 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd\" (UID: \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:19:00.771058 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.771031 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-metrics-certs\") pod \"maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd\" (UID: \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:19:00.798222 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:00.798184 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j4rk\" (UniqueName: \"kubernetes.io/projected/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-kube-api-access-6j4rk\") pod \"maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd\" (UID: \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:19:01.275005 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:01.274968 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-maas-api-tls\") pod \"maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd\" (UID: \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:19:01.277458 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:01.277434 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-maas-api-tls\") pod \"maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd\" (UID: \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:19:01.577872 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:01.577773 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:19:01.909371 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:01.909340 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd"] Oct 30 13:19:01.912027 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:19:01.911999 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0de2c5_3253_402e_99ec_70d8fb93a8e8.slice/crio-8c3d514dcf59a827f3488a8e8c64355bafde64fa9da828359807efd7dbc45c78 WatchSource:0}: Error finding container 8c3d514dcf59a827f3488a8e8c64355bafde64fa9da828359807efd7dbc45c78: Status 404 returned error can't find the container with id 8c3d514dcf59a827f3488a8e8c64355bafde64fa9da828359807efd7dbc45c78 Oct 30 13:19:02.203592 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:02.203464 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" event={"ID":"3f0de2c5-3253-402e-99ec-70d8fb93a8e8","Type":"ContainerStarted","Data":"8c3d514dcf59a827f3488a8e8c64355bafde64fa9da828359807efd7dbc45c78"} Oct 30 13:19:02.965236 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:02.965208 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:19:03.036458 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.036426 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-workload-socket\") pod \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " Oct 30 13:19:03.036697 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.036544 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-token\" (UniqueName: \"kubernetes.io/projected/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-token\") pod \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " Oct 30 13:19:03.036776 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.036676 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-podinfo\") pod \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " Oct 30 13:19:03.036841 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.036795 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-workload-socket" pod "c0a3ed5e-a26f-4514-921d-24b409b1dc27" (UID: "c0a3ed5e-a26f-4514-921d-24b409b1dc27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:03.036897 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.036847 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a3ed5e-a26f-4514-921d-24b409b1dc27-kube-api-access-tdmxf\" (UniqueName: \"kubernetes.io/projected/c0a3ed5e-a26f-4514-921d-24b409b1dc27-kube-api-access-tdmxf\") pod \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " Oct 30 13:19:03.036953 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.036933 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istiod-ca-cert\") pod \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " Oct 30 13:19:03.037007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.036963 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-envoy\") pod \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " Oct 30 13:19:03.038033 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.037410 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istiod-ca-cert" pod "c0a3ed5e-a26f-4514-921d-24b409b1dc27" (UID: "c0a3ed5e-a26f-4514-921d-24b409b1dc27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:19:03.038033 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.037480 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-credential-socket\") pod \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " Oct 30 13:19:03.038033 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.037532 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-data\") pod \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " Oct 30 13:19:03.038033 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.037563 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-workload-certs\") pod \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\" (UID: \"c0a3ed5e-a26f-4514-921d-24b409b1dc27\") " Oct 30 13:19:03.038033 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.038025 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-credential-socket" pod "c0a3ed5e-a26f-4514-921d-24b409b1dc27" (UID: "c0a3ed5e-a26f-4514-921d-24b409b1dc27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:03.038388 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.038187 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-data" pod "c0a3ed5e-a26f-4514-921d-24b409b1dc27" (UID: "c0a3ed5e-a26f-4514-921d-24b409b1dc27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:03.038388 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.038321 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-workload-certs" pod "c0a3ed5e-a26f-4514-921d-24b409b1dc27" (UID: "c0a3ed5e-a26f-4514-921d-24b409b1dc27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:03.039017 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.038924 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:19:03.039017 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.038952 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:19:03.039017 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.038968 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:19:03.039017 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.038981 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:19:03.039017 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.038996 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:19:03.040708 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.039998 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-token" pod "c0a3ed5e-a26f-4514-921d-24b409b1dc27" (UID: "c0a3ed5e-a26f-4514-921d-24b409b1dc27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:03.043437 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.043406 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a3ed5e-a26f-4514-921d-24b409b1dc27-kube-api-access-tdmxf" pod "c0a3ed5e-a26f-4514-921d-24b409b1dc27" (UID: "c0a3ed5e-a26f-4514-921d-24b409b1dc27"). InnerVolumeSpecName "kube-api-access-tdmxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:19:03.045467 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.045443 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-envoy" pod "c0a3ed5e-a26f-4514-921d-24b409b1dc27" (UID: "c0a3ed5e-a26f-4514-921d-24b409b1dc27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:19:03.053637 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.053610 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-podinfo" pod "c0a3ed5e-a26f-4514-921d-24b409b1dc27" (UID: "c0a3ed5e-a26f-4514-921d-24b409b1dc27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:19:03.140522 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.140457 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:19:03.140522 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.140481 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:19:03.140765 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.140533 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdmxf\" (UniqueName: \"kubernetes.io/projected/c0a3ed5e-a26f-4514-921d-24b409b1dc27-kube-api-access-tdmxf\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:19:03.140765 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.140544 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0a3ed5e-a26f-4514-921d-24b409b1dc27-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:19:03.211546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.211437 7010 generic.go:356] "Generic (PLEG): container finished" podID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" containerID="35aafcc6a759bb96be6d83de18da24367b5473bf0e623a0f02cfd7820d9bb0f4" exitCode=0 Oct 30 13:19:03.211546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.211538 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" Oct 30 13:19:03.211782 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.211563 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" event={"ID":"c0a3ed5e-a26f-4514-921d-24b409b1dc27","Type":"ContainerDied","Data":"35aafcc6a759bb96be6d83de18da24367b5473bf0e623a0f02cfd7820d9bb0f4"} Oct 30 13:19:03.211782 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.211628 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb" event={"ID":"c0a3ed5e-a26f-4514-921d-24b409b1dc27","Type":"ContainerDied","Data":"7a430f7c2931c1be0811c4c0afef6503fcaccfa0fc0867bc5f92d955a499eb93"} Oct 30 13:19:03.211782 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.211651 7010 scope.go:122] "RemoveContainer" containerID="35aafcc6a759bb96be6d83de18da24367b5473bf0e623a0f02cfd7820d9bb0f4" Oct 30 13:19:03.213942 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.213915 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" event={"ID":"3f0de2c5-3253-402e-99ec-70d8fb93a8e8","Type":"ContainerStarted","Data":"a2e09a2f4ed9cab397a5f512584888347584d56f17e30d1c2133162264239399"} Oct 30 13:19:03.214224 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.214204 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:19:03.224152 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.224131 7010 scope.go:122] "RemoveContainer" containerID="bb1e7b1a7e514b51b265d1829973e1c40f07091431f97f892bd5d5df44849fe0" Oct 30 13:19:03.235379 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.235354 7010 scope.go:122] "RemoveContainer" containerID="35aafcc6a759bb96be6d83de18da24367b5473bf0e623a0f02cfd7820d9bb0f4" Oct 30 13:19:03.235928 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:19:03.235901 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35aafcc6a759bb96be6d83de18da24367b5473bf0e623a0f02cfd7820d9bb0f4\": container with ID starting with 35aafcc6a759bb96be6d83de18da24367b5473bf0e623a0f02cfd7820d9bb0f4 not found: ID does not exist" containerID="35aafcc6a759bb96be6d83de18da24367b5473bf0e623a0f02cfd7820d9bb0f4" Oct 30 13:19:03.236040 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.235940 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35aafcc6a759bb96be6d83de18da24367b5473bf0e623a0f02cfd7820d9bb0f4"} err="failed to get container status \"35aafcc6a759bb96be6d83de18da24367b5473bf0e623a0f02cfd7820d9bb0f4\": rpc error: code = NotFound desc = could not find container \"35aafcc6a759bb96be6d83de18da24367b5473bf0e623a0f02cfd7820d9bb0f4\": container with ID starting with 35aafcc6a759bb96be6d83de18da24367b5473bf0e623a0f02cfd7820d9bb0f4 not found: ID does not exist" Oct 30 13:19:03.236040 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.235964 7010 scope.go:122] "RemoveContainer" containerID="bb1e7b1a7e514b51b265d1829973e1c40f07091431f97f892bd5d5df44849fe0" Oct 30 13:19:03.236311 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:19:03.236279 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb1e7b1a7e514b51b265d1829973e1c40f07091431f97f892bd5d5df44849fe0\": container with ID starting with bb1e7b1a7e514b51b265d1829973e1c40f07091431f97f892bd5d5df44849fe0 not found: ID does not exist" containerID="bb1e7b1a7e514b51b265d1829973e1c40f07091431f97f892bd5d5df44849fe0" Oct 30 13:19:03.236450 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.236318 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1e7b1a7e514b51b265d1829973e1c40f07091431f97f892bd5d5df44849fe0"} err="failed to get container status \"bb1e7b1a7e514b51b265d1829973e1c40f07091431f97f892bd5d5df44849fe0\": rpc error: code = NotFound desc = could not find container \"bb1e7b1a7e514b51b265d1829973e1c40f07091431f97f892bd5d5df44849fe0\": container with ID starting with bb1e7b1a7e514b51b265d1829973e1c40f07091431f97f892bd5d5df44849fe0 not found: ID does not exist" Oct 30 13:19:03.261020 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.260970 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw"] Oct 30 13:19:03.261761 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.261736 7010 cpu_manager.go:414] "containerMap: removing container" podUID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" containerName="istio-proxy" Oct 30 13:19:03.261761 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.261760 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" containerName="istio-proxy" Oct 30 13:19:03.261924 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.261784 7010 cpu_manager.go:414] "containerMap: removing container" podUID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" containerName="istio-proxy" Oct 30 13:19:03.261924 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.261792 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" containerName="istio-proxy" Oct 30 13:19:03.262022 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.262007 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" containerName="istio-proxy" Oct 30 13:19:03.262066 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.262025 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" containerName="istio-proxy" Oct 30 13:19:03.267363 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.267342 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" Oct 30 13:19:03.294344 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.294291 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb"] Oct 30 13:19:03.303567 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.303537 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8af0fe-openshift-default-56cbd949fc-7kfdb"] Oct 30 13:19:03.303758 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.303737 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw"] Oct 30 13:19:03.325539 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.325474 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" podStartSLOduration=3.000821766 podStartE2EDuration="3.32546168s" totalImagesPullingTime="324.639914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:03.325186974 +0000 UTC m=+1599.957374139" watchObservedRunningTime="2026-10-30 13:19:03.32546168 +0000 UTC m=+1599.957648842" Oct 30 13:19:03.334891 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.334860 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ddb271-dockercfg-gqptj\"" Oct 30 13:19:03.344190 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.344156 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxmrf\" (UniqueName: \"kubernetes.io/projected/7671398e-bfd7-4774-ac0e-eacef202eab2-kube-api-access-wxmrf\") pod \"payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw\" (UID: \"7671398e-bfd7-4774-ac0e-eacef202eab2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" Oct 30 13:19:03.344332 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.344299 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7671398e-bfd7-4774-ac0e-eacef202eab2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw\" (UID: \"7671398e-bfd7-4774-ac0e-eacef202eab2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" Oct 30 13:19:03.367460 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.367431 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ddb271\"" Oct 30 13:19:03.446934 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.446885 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxmrf\" (UniqueName: \"kubernetes.io/projected/7671398e-bfd7-4774-ac0e-eacef202eab2-kube-api-access-wxmrf\") pod \"payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw\" (UID: \"7671398e-bfd7-4774-ac0e-eacef202eab2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" Oct 30 13:19:03.447126 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.447004 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7671398e-bfd7-4774-ac0e-eacef202eab2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw\" (UID: \"7671398e-bfd7-4774-ac0e-eacef202eab2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" Oct 30 13:19:03.448198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.448006 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7671398e-bfd7-4774-ac0e-eacef202eab2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw\" (UID: \"7671398e-bfd7-4774-ac0e-eacef202eab2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" Oct 30 13:19:03.523574 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.509856 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxmrf\" (UniqueName: \"kubernetes.io/projected/7671398e-bfd7-4774-ac0e-eacef202eab2-kube-api-access-wxmrf\") pod \"payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw\" (UID: \"7671398e-bfd7-4774-ac0e-eacef202eab2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" Oct 30 13:19:03.561512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.559169 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a3ed5e-a26f-4514-921d-24b409b1dc27" path="/var/lib/kubelet/pods/c0a3ed5e-a26f-4514-921d-24b409b1dc27/volumes" Oct 30 13:19:03.598084 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.597810 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" Oct 30 13:19:03.926549 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:03.926479 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw"] Oct 30 13:19:03.930694 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:19:03.930660 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7671398e_bfd7_4774_ac0e_eacef202eab2.slice/crio-a9a79a2787534f61ef4bd53508ea0df68f367facbee5c77dc7c452300fbbe4c6 WatchSource:0}: Error finding container a9a79a2787534f61ef4bd53508ea0df68f367facbee5c77dc7c452300fbbe4c6: Status 404 returned error can't find the container with id a9a79a2787534f61ef4bd53508ea0df68f367facbee5c77dc7c452300fbbe4c6 Oct 30 13:19:04.222884 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:04.222794 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" event={"ID":"7671398e-bfd7-4774-ac0e-eacef202eab2","Type":"ContainerStarted","Data":"7425f392dd65af14e350be65d5f2aef2abbf7e07134e59de595c0312b17b7311"} Oct 30 13:19:04.222884 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:04.222852 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" event={"ID":"7671398e-bfd7-4774-ac0e-eacef202eab2","Type":"ContainerStarted","Data":"a9a79a2787534f61ef4bd53508ea0df68f367facbee5c77dc7c452300fbbe4c6"} Oct 30 13:19:04.223323 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:04.223056 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" Oct 30 13:19:04.310134 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:04.310068 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" podStartSLOduration=1.310047716 podStartE2EDuration="1.310047716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:04.307759506 +0000 UTC m=+1600.939946668" watchObservedRunningTime="2026-10-30 13:19:04.310047716 +0000 UTC m=+1600.942234879" Oct 30 13:19:06.589047 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:06.589010 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9"] Oct 30 13:19:06.593497 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:06.593463 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" Oct 30 13:19:06.611771 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:06.611740 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9"] Oct 30 13:19:06.620656 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:06.620628 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-31721c\"" Oct 30 13:19:06.699798 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:06.699756 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z6b7\" (UniqueName: \"kubernetes.io/projected/fdb4594d-71b9-405c-b1eb-654a082160e8-kube-api-access-4z6b7\") pod \"payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9\" (UID: \"fdb4594d-71b9-405c-b1eb-654a082160e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" Oct 30 13:19:06.700018 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:06.699996 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb4594d-71b9-405c-b1eb-654a082160e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9\" (UID: \"fdb4594d-71b9-405c-b1eb-654a082160e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" Oct 30 13:19:06.805454 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:06.804281 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb4594d-71b9-405c-b1eb-654a082160e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9\" (UID: \"fdb4594d-71b9-405c-b1eb-654a082160e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" Oct 30 13:19:06.805875 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:06.805851 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z6b7\" (UniqueName: \"kubernetes.io/projected/fdb4594d-71b9-405c-b1eb-654a082160e8-kube-api-access-4z6b7\") pod \"payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9\" (UID: \"fdb4594d-71b9-405c-b1eb-654a082160e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" Oct 30 13:19:06.806526 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:06.805420 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb4594d-71b9-405c-b1eb-654a082160e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9\" (UID: \"fdb4594d-71b9-405c-b1eb-654a082160e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" Oct 30 13:19:06.843625 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:06.833538 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z6b7\" (UniqueName: \"kubernetes.io/projected/fdb4594d-71b9-405c-b1eb-654a082160e8-kube-api-access-4z6b7\") pod \"payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9\" (UID: \"fdb4594d-71b9-405c-b1eb-654a082160e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" Oct 30 13:19:06.911305 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:06.911268 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" Oct 30 13:19:07.225985 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:07.225518 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9"] Oct 30 13:19:07.241114 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:19:07.241072 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb4594d_71b9_405c_b1eb_654a082160e8.slice/crio-625e98846aef128e04014e19ac381d8cdac621e3f2bf22b2580081e188a3dd3b WatchSource:0}: Error finding container 625e98846aef128e04014e19ac381d8cdac621e3f2bf22b2580081e188a3dd3b: Status 404 returned error can't find the container with id 625e98846aef128e04014e19ac381d8cdac621e3f2bf22b2580081e188a3dd3b Oct 30 13:19:07.250510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:07.250475 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" event={"ID":"fdb4594d-71b9-405c-b1eb-654a082160e8","Type":"ContainerStarted","Data":"625e98846aef128e04014e19ac381d8cdac621e3f2bf22b2580081e188a3dd3b"} Oct 30 13:19:08.258433 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:08.258396 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" event={"ID":"fdb4594d-71b9-405c-b1eb-654a082160e8","Type":"ContainerStarted","Data":"d26f2d777f13135af61b7b065022bdaca17b578d66ef60fb5508e537d1d2c5e7"} Oct 30 13:19:08.258944 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:08.258464 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" Oct 30 13:19:08.306432 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:08.306375 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" podStartSLOduration=2.306361968 podStartE2EDuration="2.306361968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:08.303014501 +0000 UTC m=+1604.935201662" watchObservedRunningTime="2026-10-30 13:19:08.306361968 +0000 UTC m=+1604.938549134" Oct 30 13:19:09.228873 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:09.228842 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:19:15.240860 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:15.240829 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" Oct 30 13:19:19.273151 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:19.273123 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" Oct 30 13:19:29.401873 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:19:29.401835 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:19:37.649019 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:37.648907 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm"] Oct 30 13:19:37.654771 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:37.654749 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:19:37.673457 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:37.673414 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm"] Oct 30 13:19:37.684906 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:37.684874 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-31721c-serving-cert\"" Oct 30 13:19:37.799628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:37.799591 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhmc8\" (UniqueName: \"kubernetes.io/projected/a153c1e7-067c-426b-90df-87fa6649d078-kube-api-access-fhmc8\") pod \"maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm\" (UID: \"a153c1e7-067c-426b-90df-87fa6649d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:19:37.799811 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:37.799741 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a153c1e7-067c-426b-90df-87fa6649d078-maas-api-tls\") pod \"maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm\" (UID: \"a153c1e7-067c-426b-90df-87fa6649d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:19:37.799863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:37.799846 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a153c1e7-067c-426b-90df-87fa6649d078-openshift-service-ca\") pod \"maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm\" (UID: \"a153c1e7-067c-426b-90df-87fa6649d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:19:37.799906 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:37.799882 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a153c1e7-067c-426b-90df-87fa6649d078-metrics-certs\") pod \"maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm\" (UID: \"a153c1e7-067c-426b-90df-87fa6649d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:19:37.901747 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:37.901667 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhmc8\" (UniqueName: \"kubernetes.io/projected/a153c1e7-067c-426b-90df-87fa6649d078-kube-api-access-fhmc8\") pod \"maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm\" (UID: \"a153c1e7-067c-426b-90df-87fa6649d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:19:37.901905 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:37.901765 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a153c1e7-067c-426b-90df-87fa6649d078-maas-api-tls\") pod \"maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm\" (UID: \"a153c1e7-067c-426b-90df-87fa6649d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:19:37.901905 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:37.901815 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a153c1e7-067c-426b-90df-87fa6649d078-openshift-service-ca\") pod \"maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm\" (UID: \"a153c1e7-067c-426b-90df-87fa6649d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:19:37.901905 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:37.901840 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a153c1e7-067c-426b-90df-87fa6649d078-metrics-certs\") pod \"maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm\" (UID: \"a153c1e7-067c-426b-90df-87fa6649d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:19:37.902513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:37.902473 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a153c1e7-067c-426b-90df-87fa6649d078-openshift-service-ca\") pod \"maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm\" (UID: \"a153c1e7-067c-426b-90df-87fa6649d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:19:37.904473 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:37.904439 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a153c1e7-067c-426b-90df-87fa6649d078-maas-api-tls\") pod \"maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm\" (UID: \"a153c1e7-067c-426b-90df-87fa6649d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:19:37.904609 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:37.904443 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a153c1e7-067c-426b-90df-87fa6649d078-metrics-certs\") pod \"maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm\" (UID: \"a153c1e7-067c-426b-90df-87fa6649d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:19:37.925741 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:37.925709 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhmc8\" (UniqueName: \"kubernetes.io/projected/a153c1e7-067c-426b-90df-87fa6649d078-kube-api-access-fhmc8\") pod \"maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm\" (UID: \"a153c1e7-067c-426b-90df-87fa6649d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:19:37.976850 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:37.976812 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:19:38.298476 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:38.298440 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm"] Oct 30 13:19:38.299988 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:19:38.299961 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda153c1e7_067c_426b_90df_87fa6649d078.slice/crio-0e2492f8f0c948910e9e548bdc51e8ff536039d11dffaf2b2c2f0d1e6a93816e WatchSource:0}: Error finding container 0e2492f8f0c948910e9e548bdc51e8ff536039d11dffaf2b2c2f0d1e6a93816e: Status 404 returned error can't find the container with id 0e2492f8f0c948910e9e548bdc51e8ff536039d11dffaf2b2c2f0d1e6a93816e Oct 30 13:19:38.469957 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:38.469920 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" event={"ID":"a153c1e7-067c-426b-90df-87fa6649d078","Type":"ContainerStarted","Data":"0e2492f8f0c948910e9e548bdc51e8ff536039d11dffaf2b2c2f0d1e6a93816e"} Oct 30 13:19:39.479079 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:39.479037 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" event={"ID":"a153c1e7-067c-426b-90df-87fa6649d078","Type":"ContainerStarted","Data":"5c41d604704cf084a0769f96cfacc9133fd618876aa5ca2f114c79c9ff1114b8"} Oct 30 13:19:39.479540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:39.479112 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:19:39.522671 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:39.522620 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" podStartSLOduration=2.077274695 podStartE2EDuration="2.522601316s" totalImagesPullingTime="445.326621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:39.522388242 +0000 UTC m=+1636.154575405" watchObservedRunningTime="2026-10-30 13:19:39.522601316 +0000 UTC m=+1636.154788478" Oct 30 13:19:45.491199 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:19:45.491169 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:19:59.401280 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:19:59.401235 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:20:03.781140 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:03.781103 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l"] Oct 30 13:20:03.781723 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:03.781323 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" podUID="1fe92cf8-8670-47c1-bff4-604674fd22cb" containerName="maas-api" containerID="cri-o://24d47aac89ab06e4f038ef9fc2b50f34095f1835dd9a58bc970c89ab07e135e2" gracePeriod=30 Oct 30 13:20:04.188399 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.188366 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:20:04.415605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.403330 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fe92cf8-8670-47c1-bff4-604674fd22cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fe92cf8-8670-47c1-bff4-604674fd22cb-metrics-certs\") pod \"1fe92cf8-8670-47c1-bff4-604674fd22cb\" (UID: \"1fe92cf8-8670-47c1-bff4-604674fd22cb\") " Oct 30 13:20:04.415605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.403583 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe92cf8-8670-47c1-bff4-604674fd22cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe92cf8-8670-47c1-bff4-604674fd22cb-openshift-service-ca\") pod \"1fe92cf8-8670-47c1-bff4-604674fd22cb\" (UID: \"1fe92cf8-8670-47c1-bff4-604674fd22cb\") " Oct 30 13:20:04.415605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.403700 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe92cf8-8670-47c1-bff4-604674fd22cb-kube-api-access-qx5kv\" (UniqueName: \"kubernetes.io/projected/1fe92cf8-8670-47c1-bff4-604674fd22cb-kube-api-access-qx5kv\") pod \"1fe92cf8-8670-47c1-bff4-604674fd22cb\" (UID: \"1fe92cf8-8670-47c1-bff4-604674fd22cb\") " Oct 30 13:20:04.415605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.403760 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fe92cf8-8670-47c1-bff4-604674fd22cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fe92cf8-8670-47c1-bff4-604674fd22cb-maas-api-tls\") pod \"1fe92cf8-8670-47c1-bff4-604674fd22cb\" (UID: \"1fe92cf8-8670-47c1-bff4-604674fd22cb\") " Oct 30 13:20:04.415605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.404635 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe92cf8-8670-47c1-bff4-604674fd22cb-openshift-service-ca" pod "1fe92cf8-8670-47c1-bff4-604674fd22cb" (UID: "1fe92cf8-8670-47c1-bff4-604674fd22cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:20:04.440830 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.438127 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe92cf8-8670-47c1-bff4-604674fd22cb-metrics-certs" pod "1fe92cf8-8670-47c1-bff4-604674fd22cb" (UID: "1fe92cf8-8670-47c1-bff4-604674fd22cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:20:04.440830 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.438203 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe92cf8-8670-47c1-bff4-604674fd22cb-kube-api-access-qx5kv" pod "1fe92cf8-8670-47c1-bff4-604674fd22cb" (UID: "1fe92cf8-8670-47c1-bff4-604674fd22cb"). InnerVolumeSpecName "kube-api-access-qx5kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:20:04.440830 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.439360 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe92cf8-8670-47c1-bff4-604674fd22cb-maas-api-tls" pod "1fe92cf8-8670-47c1-bff4-604674fd22cb" (UID: "1fe92cf8-8670-47c1-bff4-604674fd22cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:20:04.507031 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.506695 7010 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe92cf8-8670-47c1-bff4-604674fd22cb-openshift-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:20:04.507031 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.506766 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx5kv\" (UniqueName: \"kubernetes.io/projected/1fe92cf8-8670-47c1-bff4-604674fd22cb-kube-api-access-qx5kv\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:20:04.507031 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.506783 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fe92cf8-8670-47c1-bff4-604674fd22cb-maas-api-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:20:04.507031 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.506798 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fe92cf8-8670-47c1-bff4-604674fd22cb-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:20:04.695261 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.695172 7010 generic.go:356] "Generic (PLEG): container finished" podID="1fe92cf8-8670-47c1-bff4-604674fd22cb" containerID="24d47aac89ab06e4f038ef9fc2b50f34095f1835dd9a58bc970c89ab07e135e2" exitCode=0 Oct 30 13:20:04.695261 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.695213 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" event={"ID":"1fe92cf8-8670-47c1-bff4-604674fd22cb","Type":"ContainerDied","Data":"24d47aac89ab06e4f038ef9fc2b50f34095f1835dd9a58bc970c89ab07e135e2"} Oct 30 13:20:04.695261 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.695232 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" Oct 30 13:20:04.695261 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.695258 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l" event={"ID":"1fe92cf8-8670-47c1-bff4-604674fd22cb","Type":"ContainerDied","Data":"40914f60401283b33f280cec3662670e390230c95c35ab8f6ae9f7eff5b0fb85"} Oct 30 13:20:04.695578 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.695278 7010 scope.go:122] "RemoveContainer" containerID="24d47aac89ab06e4f038ef9fc2b50f34095f1835dd9a58bc970c89ab07e135e2" Oct 30 13:20:04.707162 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.706844 7010 scope.go:122] "RemoveContainer" containerID="24d47aac89ab06e4f038ef9fc2b50f34095f1835dd9a58bc970c89ab07e135e2" Oct 30 13:20:04.707293 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:20:04.707196 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d47aac89ab06e4f038ef9fc2b50f34095f1835dd9a58bc970c89ab07e135e2\": container with ID starting with 24d47aac89ab06e4f038ef9fc2b50f34095f1835dd9a58bc970c89ab07e135e2 not found: ID does not exist" containerID="24d47aac89ab06e4f038ef9fc2b50f34095f1835dd9a58bc970c89ab07e135e2" Oct 30 13:20:04.707293 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.707226 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d47aac89ab06e4f038ef9fc2b50f34095f1835dd9a58bc970c89ab07e135e2"} err="failed to get container status \"24d47aac89ab06e4f038ef9fc2b50f34095f1835dd9a58bc970c89ab07e135e2\": rpc error: code = NotFound desc = could not find container \"24d47aac89ab06e4f038ef9fc2b50f34095f1835dd9a58bc970c89ab07e135e2\": container with ID starting with 24d47aac89ab06e4f038ef9fc2b50f34095f1835dd9a58bc970c89ab07e135e2 not found: ID does not exist" Oct 30 13:20:04.772643 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.771866 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l"] Oct 30 13:20:04.780412 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:04.780194 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4644a1-74d9f89479-27l8l"] Oct 30 13:20:05.491577 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:05.491542 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe92cf8-8670-47c1-bff4-604674fd22cb" path="/var/lib/kubelet/pods/1fe92cf8-8670-47c1-bff4-604674fd22cb/volumes" Oct 30 13:20:25.776135 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.776096 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j"] Oct 30 13:20:25.777148 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.777123 7010 cpu_manager.go:414] "containerMap: removing container" podUID="1fe92cf8-8670-47c1-bff4-604674fd22cb" containerName="maas-api" Oct 30 13:20:25.777225 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.777152 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe92cf8-8670-47c1-bff4-604674fd22cb" containerName="maas-api" Oct 30 13:20:25.777330 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.777313 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe92cf8-8670-47c1-bff4-604674fd22cb" containerName="maas-api" Oct 30 13:20:25.782999 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.782974 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:25.806402 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.806368 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j"] Oct 30 13:20:25.815303 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.815276 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5dd8cc-openshift-default-dockercfg-qzm98\"" Oct 30 13:20:25.873042 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.873005 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z2wt\" (UniqueName: \"kubernetes.io/projected/51645cac-ccad-44c3-9928-d0283dc273ee-kube-api-access-6z2wt\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:25.873237 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.873114 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51645cac-ccad-44c3-9928-d0283dc273ee-istio-podinfo\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:25.873237 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.873211 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-istio-data\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:25.873368 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.873314 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-workload-socket\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:25.873368 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.873349 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51645cac-ccad-44c3-9928-d0283dc273ee-istio-token\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:25.873549 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.873532 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-workload-certs\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:25.873626 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.873602 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51645cac-ccad-44c3-9928-d0283dc273ee-istiod-ca-cert\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:25.873755 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.873736 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-credential-socket\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:25.873816 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.873778 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-istio-envoy\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:25.977041 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.976989 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z2wt\" (UniqueName: \"kubernetes.io/projected/51645cac-ccad-44c3-9928-d0283dc273ee-kube-api-access-6z2wt\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:25.977859 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.977813 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51645cac-ccad-44c3-9928-d0283dc273ee-istio-podinfo\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:25.978705 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.978684 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-istio-data\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:25.979943 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.979691 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-workload-socket\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:25.990957 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.979557 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-istio-data\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:25.996082 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.996052 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-workload-socket\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:26.006627 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:25.990902 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51645cac-ccad-44c3-9928-d0283dc273ee-istio-token\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:26.036764 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.036682 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-workload-certs\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:26.037641 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.037618 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51645cac-ccad-44c3-9928-d0283dc273ee-istiod-ca-cert\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:26.038887 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.038607 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51645cac-ccad-44c3-9928-d0283dc273ee-istiod-ca-cert\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:26.039057 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.038632 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-workload-certs\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:26.039347 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.039330 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-credential-socket\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:26.057845 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.054349 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z2wt\" (UniqueName: \"kubernetes.io/projected/51645cac-ccad-44c3-9928-d0283dc273ee-kube-api-access-6z2wt\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:26.058538 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.058512 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51645cac-ccad-44c3-9928-d0283dc273ee-istio-podinfo\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:26.058778 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.058670 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-istio-envoy\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:26.059317 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.059294 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-credential-socket\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:26.069520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.067170 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51645cac-ccad-44c3-9928-d0283dc273ee-istio-token\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:26.094676 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.094601 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-istio-envoy\") pod \"e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:26.107519 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.101770 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:26.519212 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.519180 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j"] Oct 30 13:20:26.523684 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:20:26.523653 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51645cac_ccad_44c3_9928_d0283dc273ee.slice/crio-15a87e568e64943d4d1be45233977515d1c7be57b33528147d95d62a48834e02 WatchSource:0}: Error finding container 15a87e568e64943d4d1be45233977515d1c7be57b33528147d95d62a48834e02: Status 404 returned error can't find the container with id 15a87e568e64943d4d1be45233977515d1c7be57b33528147d95d62a48834e02 Oct 30 13:20:26.526767 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.526732 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:26.526873 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.526816 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:26.526873 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.526862 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:20:26.861100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.861009 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" event={"ID":"51645cac-ccad-44c3-9928-d0283dc273ee","Type":"ContainerStarted","Data":"203d983a17ecd15c8e8f0f201bc685f37df111f4713ea8702c50ec0cd7e3cfc9"} Oct 30 13:20:26.861100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.861062 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" event={"ID":"51645cac-ccad-44c3-9928-d0283dc273ee","Type":"ContainerStarted","Data":"15a87e568e64943d4d1be45233977515d1c7be57b33528147d95d62a48834e02"} Oct 30 13:20:26.920975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:26.920903 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" podStartSLOduration=1.920886906 podStartE2EDuration="1.920886906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:26.920319683 +0000 UTC m=+1683.552506872" watchObservedRunningTime="2026-10-30 13:20:26.920886906 +0000 UTC m=+1683.553074069" Oct 30 13:20:27.108240 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:27.108157 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:27.117621 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:27.116597 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:27.893538 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:27.890953 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:27.893538 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:27.892071 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:20:29.408621 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:20:29.408585 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:20:51.445396 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.445358 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg"] Oct 30 13:20:51.449756 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.449732 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.470935 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.470897 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg"] Oct 30 13:20:51.477770 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.477738 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ddb271\"/\"test-model-ddb271-kserve-self-signed-certs\"" Oct 30 13:20:51.477957 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.477815 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ddb271\"/\"default-dockercfg-wrmrl\"" Oct 30 13:20:51.481238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.481206 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ddb271\"/\"kube-root-ca.crt\"" Oct 30 13:20:51.481404 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.481218 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ddb271\"/\"openshift-service-ca.crt\"" Oct 30 13:20:51.543509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.542118 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-home\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.543509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.542304 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-tmp-dir\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.543509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.542581 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-dshm\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.543509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.542718 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96f80372-7a4d-464d-a85b-ad4fa0742324-tls-certs\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.543509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.542752 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-model-cache\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.543509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.542783 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cth9g\" (UniqueName: \"kubernetes.io/projected/96f80372-7a4d-464d-a85b-ad4fa0742324-kube-api-access-cth9g\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.644100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.644046 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-dshm\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.644386 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.644346 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96f80372-7a4d-464d-a85b-ad4fa0742324-tls-certs\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.644504 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.644397 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-model-cache\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.644504 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.644432 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cth9g\" (UniqueName: \"kubernetes.io/projected/96f80372-7a4d-464d-a85b-ad4fa0742324-kube-api-access-cth9g\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.644652 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.644629 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-home\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.645057 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.644727 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-tmp-dir\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.645057 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.644749 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-model-cache\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.645057 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.645007 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-tmp-dir\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.645057 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.645053 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-home\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.647042 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.647016 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-dshm\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.647403 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.647380 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96f80372-7a4d-464d-a85b-ad4fa0742324-tls-certs\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.669545 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.669508 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cth9g\" (UniqueName: \"kubernetes.io/projected/96f80372-7a4d-464d-a85b-ad4fa0742324-kube-api-access-cth9g\") pod \"test-model-ddb271-kserve-56f49859cf-dzxlg\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:51.776440 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:51.776149 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:52.202555 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:20:52.193192 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96f80372_7a4d_464d_a85b_ad4fa0742324.slice/crio-01d7d6b8a5639288a2604da9348f65d69f1fdcf206cc8d5a6bcb04a71ee54748 WatchSource:0}: Error finding container 01d7d6b8a5639288a2604da9348f65d69f1fdcf206cc8d5a6bcb04a71ee54748: Status 404 returned error can't find the container with id 01d7d6b8a5639288a2604da9348f65d69f1fdcf206cc8d5a6bcb04a71ee54748 Oct 30 13:20:52.214513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:52.206388 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg"] Oct 30 13:20:53.096338 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:53.096300 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" event={"ID":"96f80372-7a4d-464d-a85b-ad4fa0742324","Type":"ContainerStarted","Data":"c41c42b1a2a1290999488e99999ac7653932761fda6cb1d94d144057dcc68cec"} Oct 30 13:20:53.096338 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:53.096355 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" event={"ID":"96f80372-7a4d-464d-a85b-ad4fa0742324","Type":"ContainerStarted","Data":"01d7d6b8a5639288a2604da9348f65d69f1fdcf206cc8d5a6bcb04a71ee54748"} Oct 30 13:20:53.141157 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:53.141090 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" podStartSLOduration=2.141068559 podStartE2EDuration="2.141068559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:53.137910731 +0000 UTC m=+1709.770097893" watchObservedRunningTime="2026-10-30 13:20:53.141068559 +0000 UTC m=+1709.773255722" Oct 30 13:20:53.777199 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:53.777160 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:20:55.818079 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:55.818046 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6"] Oct 30 13:20:55.822598 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:55.822569 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" Oct 30 13:20:55.838565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:55.838484 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-8d82d5\"" Oct 30 13:20:55.851246 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:55.851203 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-8d82d5-dockercfg-46hff\"" Oct 30 13:20:55.854543 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:55.854518 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6"] Oct 30 13:20:55.970512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:55.968537 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246d0d04-97d3-4762-9075-777053be1de9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6\" (UID: \"246d0d04-97d3-4762-9075-777053be1de9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" Oct 30 13:20:55.970512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:55.968872 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfw9q\" (UniqueName: \"kubernetes.io/projected/246d0d04-97d3-4762-9075-777053be1de9-kube-api-access-kfw9q\") pod \"payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6\" (UID: \"246d0d04-97d3-4762-9075-777053be1de9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" Oct 30 13:20:56.071925 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:56.071799 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfw9q\" (UniqueName: \"kubernetes.io/projected/246d0d04-97d3-4762-9075-777053be1de9-kube-api-access-kfw9q\") pod \"payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6\" (UID: \"246d0d04-97d3-4762-9075-777053be1de9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" Oct 30 13:20:56.072222 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:56.072201 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246d0d04-97d3-4762-9075-777053be1de9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6\" (UID: \"246d0d04-97d3-4762-9075-777053be1de9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" Oct 30 13:20:56.073251 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:56.073225 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246d0d04-97d3-4762-9075-777053be1de9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6\" (UID: \"246d0d04-97d3-4762-9075-777053be1de9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" Oct 30 13:20:56.101416 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:56.101377 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfw9q\" (UniqueName: \"kubernetes.io/projected/246d0d04-97d3-4762-9075-777053be1de9-kube-api-access-kfw9q\") pod \"payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6\" (UID: \"246d0d04-97d3-4762-9075-777053be1de9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" Oct 30 13:20:56.137743 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:56.137711 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" Oct 30 13:20:56.490537 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:56.490479 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6"] Oct 30 13:20:56.492342 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:20:56.492302 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246d0d04_97d3_4762_9075_777053be1de9.slice/crio-7e2d6a49479cb25e039f207c50e4136d93f674e7478c92454fcc69171c35d2e6 WatchSource:0}: Error finding container 7e2d6a49479cb25e039f207c50e4136d93f674e7478c92454fcc69171c35d2e6: Status 404 returned error can't find the container with id 7e2d6a49479cb25e039f207c50e4136d93f674e7478c92454fcc69171c35d2e6 Oct 30 13:20:57.125332 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:57.125298 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" event={"ID":"246d0d04-97d3-4762-9075-777053be1de9","Type":"ContainerStarted","Data":"0a6b54b305cb6edd985e73c919029d7d549cec5825798bf02070a3330b0b864d"} Oct 30 13:20:57.125860 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:57.125356 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" event={"ID":"246d0d04-97d3-4762-9075-777053be1de9","Type":"ContainerStarted","Data":"7e2d6a49479cb25e039f207c50e4136d93f674e7478c92454fcc69171c35d2e6"} Oct 30 13:20:57.125860 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:57.125389 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" Oct 30 13:20:57.168444 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:57.168386 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" podStartSLOduration=2.168369873 podStartE2EDuration="2.168369873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:57.167078244 +0000 UTC m=+1713.799265405" watchObservedRunningTime="2026-10-30 13:20:57.168369873 +0000 UTC m=+1713.800557037" Oct 30 13:20:59.255884 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:59.255818 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4"] Oct 30 13:20:59.280318 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:59.280276 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" Oct 30 13:20:59.296870 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:59.296840 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-09e4c0\"" Oct 30 13:20:59.313374 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:59.313344 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-09e4c0-dockercfg-scsgn\"" Oct 30 13:20:59.324041 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:59.323836 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4"] Oct 30 13:20:59.362518 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:59.361956 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3233ae-f903-495a-aa4c-1e8fe88efa5b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4\" (UID: \"5d3233ae-f903-495a-aa4c-1e8fe88efa5b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" Oct 30 13:20:59.362518 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:59.362254 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsw27\" (UniqueName: \"kubernetes.io/projected/5d3233ae-f903-495a-aa4c-1e8fe88efa5b-kube-api-access-qsw27\") pod \"payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4\" (UID: \"5d3233ae-f903-495a-aa4c-1e8fe88efa5b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" Oct 30 13:20:59.411291 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:20:59.411251 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:20:59.463410 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:59.463369 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3233ae-f903-495a-aa4c-1e8fe88efa5b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4\" (UID: \"5d3233ae-f903-495a-aa4c-1e8fe88efa5b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" Oct 30 13:20:59.463632 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:59.463599 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsw27\" (UniqueName: \"kubernetes.io/projected/5d3233ae-f903-495a-aa4c-1e8fe88efa5b-kube-api-access-qsw27\") pod \"payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4\" (UID: \"5d3233ae-f903-495a-aa4c-1e8fe88efa5b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" Oct 30 13:20:59.464141 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:59.464121 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3233ae-f903-495a-aa4c-1e8fe88efa5b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4\" (UID: \"5d3233ae-f903-495a-aa4c-1e8fe88efa5b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" Oct 30 13:20:59.487820 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:59.487786 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsw27\" (UniqueName: \"kubernetes.io/projected/5d3233ae-f903-495a-aa4c-1e8fe88efa5b-kube-api-access-qsw27\") pod \"payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4\" (UID: \"5d3233ae-f903-495a-aa4c-1e8fe88efa5b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" Oct 30 13:20:59.617003 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:59.616916 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" Oct 30 13:20:59.931270 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:20:59.931238 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4"] Oct 30 13:20:59.932752 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:20:59.932703 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3233ae_f903_495a_aa4c_1e8fe88efa5b.slice/crio-ba4efd88b1a14a4873365938ad3cabde6311e0156ba89fdd43c8900c7f9b41dc WatchSource:0}: Error finding container ba4efd88b1a14a4873365938ad3cabde6311e0156ba89fdd43c8900c7f9b41dc: Status 404 returned error can't find the container with id ba4efd88b1a14a4873365938ad3cabde6311e0156ba89fdd43c8900c7f9b41dc Oct 30 13:21:00.148971 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:00.148935 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" event={"ID":"5d3233ae-f903-495a-aa4c-1e8fe88efa5b","Type":"ContainerStarted","Data":"da03b500e73b5b9495fb9e221b8c71311828b2b1839bc2b16c66786d91858f87"} Oct 30 13:21:00.149173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:00.149000 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" Oct 30 13:21:00.149173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:00.149019 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" event={"ID":"5d3233ae-f903-495a-aa4c-1e8fe88efa5b","Type":"ContainerStarted","Data":"ba4efd88b1a14a4873365938ad3cabde6311e0156ba89fdd43c8900c7f9b41dc"} Oct 30 13:21:00.191506 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:00.191386 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" podStartSLOduration=1.191372128 podStartE2EDuration="1.191372128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:00.189861467 +0000 UTC m=+1716.822048623" watchObservedRunningTime="2026-10-30 13:21:00.191372128 +0000 UTC m=+1716.823559289" Oct 30 13:21:01.776566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:01.776531 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:21:01.789553 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:01.789522 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:21:02.174875 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:02.174840 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:21:04.317106 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.317023 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst"] Oct 30 13:21:04.321671 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.321651 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:21:04.352698 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.352438 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4651e8-serving-cert\"" Oct 30 13:21:04.363643 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.363605 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst"] Oct 30 13:21:04.440333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.440289 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mncvk\" (UniqueName: \"kubernetes.io/projected/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-kube-api-access-mncvk\") pod \"maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst\" (UID: \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:21:04.440333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.440334 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-metrics-certs\") pod \"maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst\" (UID: \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:21:04.440607 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.440432 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-maas-api-tls\") pod \"maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst\" (UID: \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:21:04.440607 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.440580 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst\" (UID: \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:21:04.542472 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.542438 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst\" (UID: \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:21:04.542682 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.542626 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mncvk\" (UniqueName: \"kubernetes.io/projected/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-kube-api-access-mncvk\") pod \"maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst\" (UID: \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:21:04.542682 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.542655 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-metrics-certs\") pod \"maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst\" (UID: \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:21:04.542803 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.542690 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-maas-api-tls\") pod \"maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst\" (UID: \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:21:04.543147 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.543120 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst\" (UID: \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:21:04.545211 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.545187 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-maas-api-tls\") pod \"maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst\" (UID: \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:21:04.545342 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.545258 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-metrics-certs\") pod \"maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst\" (UID: \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:21:04.591138 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.589920 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mncvk\" (UniqueName: \"kubernetes.io/projected/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-kube-api-access-mncvk\") pod \"maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst\" (UID: \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:21:04.635596 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.635562 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:21:04.952920 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.952889 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst"] Oct 30 13:21:04.954220 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:21:04.954194 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8b97ee_4a2c_4987_9dbe_3b1ccc0f423a.slice/crio-06854218ad3559af5f3bbd0dadb03d9be7621c5818701bf71d937409c7035bfb WatchSource:0}: Error finding container 06854218ad3559af5f3bbd0dadb03d9be7621c5818701bf71d937409c7035bfb: Status 404 returned error can't find the container with id 06854218ad3559af5f3bbd0dadb03d9be7621c5818701bf71d937409c7035bfb Oct 30 13:21:04.955840 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:04.955818 7010 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:21:05.213172 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:05.213065 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" event={"ID":"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a","Type":"ContainerStarted","Data":"06854218ad3559af5f3bbd0dadb03d9be7621c5818701bf71d937409c7035bfb"} Oct 30 13:21:06.221196 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:06.221158 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" event={"ID":"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a","Type":"ContainerStarted","Data":"4ab0dba59438fc0533b82450fe813d54d87247111e6e2afe0ccdcf3ca7eef4bb"} Oct 30 13:21:06.221693 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:06.221349 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:21:06.266116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:06.266059 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" podStartSLOduration=1.921330607 podStartE2EDuration="2.266041786s" totalImagesPullingTime="344.711179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:06.264416375 +0000 UTC m=+1722.896603533" watchObservedRunningTime="2026-10-30 13:21:06.266041786 +0000 UTC m=+1722.898228949" Oct 30 13:21:08.135392 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:08.135363 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" Oct 30 13:21:11.159341 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:11.159311 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" Oct 30 13:21:12.232725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:12.232698 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:21:25.406688 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:25.406645 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl"] Oct 30 13:21:25.407224 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:25.406913 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" podUID="03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" containerName="istio-proxy" containerID="cri-o://9af61fdb229f6152c7f3200bded503340a66729535da56c7702ace8ed3f19a3e" gracePeriod=30 Oct 30 13:21:25.735041 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:25.734923 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk"] Oct 30 13:21:25.735223 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:25.735191 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" podUID="2adf6e82-f5f4-48cc-aca3-85a34f5ec552" containerName="istio-proxy" containerID="cri-o://6ebc71d3ea9ad9fbde49c8c7c2261aa68cb9ba8948f255306901bbcc1283b863" gracePeriod=30 Oct 30 13:21:29.401600 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:21:29.401558 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:21:30.684623 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.684567 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:21:30.814603 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.814574 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-workload-certs\") pod \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " Oct 30 13:21:30.814753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.814654 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-kube-api-access-crpwz\" (UniqueName: \"kubernetes.io/projected/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-kube-api-access-crpwz\") pod \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " Oct 30 13:21:30.814753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.814684 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-podinfo\") pod \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " Oct 30 13:21:30.814875 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.814771 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-workload-socket\") pod \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " Oct 30 13:21:30.814875 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.814798 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-credential-socket\") pod \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " Oct 30 13:21:30.814875 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.814858 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-token\" (UniqueName: \"kubernetes.io/projected/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-token\") pod \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " Oct 30 13:21:30.815045 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.814892 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-workload-certs" pod "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" (UID: "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:30.815045 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.814930 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-envoy\") pod \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " Oct 30 13:21:30.815045 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.814980 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-data\") pod \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " Oct 30 13:21:30.815199 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.815108 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-credential-socket" pod "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" (UID: "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:30.815199 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.815120 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-workload-socket" pod "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" (UID: "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:30.815316 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.815297 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-data" pod "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" (UID: "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:30.815369 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.815330 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istiod-ca-cert\") pod \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\" (UID: \"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d\") " Oct 30 13:21:30.815565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.815541 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istiod-ca-cert" pod "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" (UID: "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:21:30.816139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.816113 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:30.816229 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.816144 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:30.816229 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.816159 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:30.816229 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.816172 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:30.816229 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.816186 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:30.817215 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.817176 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-envoy" pod "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" (UID: "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:30.817315 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.817291 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-kube-api-access-crpwz" pod "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" (UID: "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d"). InnerVolumeSpecName "kube-api-access-crpwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:30.817367 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.817324 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-podinfo" pod "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" (UID: "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:21:30.817410 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.817387 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-token" pod "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" (UID: "03b1816f-01c8-4c93-8ba3-ffdd5a1be50d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:30.920061 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.920024 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crpwz\" (UniqueName: \"kubernetes.io/projected/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-kube-api-access-crpwz\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:30.920197 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.920063 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:30.920197 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.920079 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:30.920197 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.920092 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:30.989791 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:30.989759 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:21:31.021753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.021721 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-data\") pod \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " Oct 30 13:21:31.021958 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.021795 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istiod-ca-cert\") pod \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " Oct 30 13:21:31.021958 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.021851 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-workload-certs\") pod \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " Oct 30 13:21:31.021958 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.021893 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-podinfo\") pod \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " Oct 30 13:21:31.022128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.022095 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-workload-socket\") pod \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " Oct 30 13:21:31.022188 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.022069 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-data" pod "2adf6e82-f5f4-48cc-aca3-85a34f5ec552" (UID: "2adf6e82-f5f4-48cc-aca3-85a34f5ec552"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:31.022188 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.022142 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-credential-socket\") pod \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " Oct 30 13:21:31.022188 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.022123 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-workload-certs" pod "2adf6e82-f5f4-48cc-aca3-85a34f5ec552" (UID: "2adf6e82-f5f4-48cc-aca3-85a34f5ec552"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:31.022339 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.022200 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istiod-ca-cert" pod "2adf6e82-f5f4-48cc-aca3-85a34f5ec552" (UID: "2adf6e82-f5f4-48cc-aca3-85a34f5ec552"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:21:31.022339 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.022223 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-kube-api-access-6sq7m\" (UniqueName: \"kubernetes.io/projected/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-kube-api-access-6sq7m\") pod \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " Oct 30 13:21:31.022339 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.022272 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-token\" (UniqueName: \"kubernetes.io/projected/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-token\") pod \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " Oct 30 13:21:31.022339 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.022296 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-workload-socket" pod "2adf6e82-f5f4-48cc-aca3-85a34f5ec552" (UID: "2adf6e82-f5f4-48cc-aca3-85a34f5ec552"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:31.022339 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.022314 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-envoy\") pod \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\" (UID: \"2adf6e82-f5f4-48cc-aca3-85a34f5ec552\") " Oct 30 13:21:31.022636 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.022415 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-credential-socket" pod "2adf6e82-f5f4-48cc-aca3-85a34f5ec552" (UID: "2adf6e82-f5f4-48cc-aca3-85a34f5ec552"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:31.023073 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.023050 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:31.023073 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.023073 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:31.023292 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.023088 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:31.023292 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.023101 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:31.023292 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.023114 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:31.024450 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.024413 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-podinfo" pod "2adf6e82-f5f4-48cc-aca3-85a34f5ec552" (UID: "2adf6e82-f5f4-48cc-aca3-85a34f5ec552"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:21:31.024641 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.024617 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-envoy" pod "2adf6e82-f5f4-48cc-aca3-85a34f5ec552" (UID: "2adf6e82-f5f4-48cc-aca3-85a34f5ec552"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:21:31.024999 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.024969 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-token" pod "2adf6e82-f5f4-48cc-aca3-85a34f5ec552" (UID: "2adf6e82-f5f4-48cc-aca3-85a34f5ec552"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:31.025079 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.025003 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-kube-api-access-6sq7m" pod "2adf6e82-f5f4-48cc-aca3-85a34f5ec552" (UID: "2adf6e82-f5f4-48cc-aca3-85a34f5ec552"). InnerVolumeSpecName "kube-api-access-6sq7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:21:31.124567 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.124535 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:31.124567 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.124563 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sq7m\" (UniqueName: \"kubernetes.io/projected/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-kube-api-access-6sq7m\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:31.124567 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.124577 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:31.124803 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.124587 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2adf6e82-f5f4-48cc-aca3-85a34f5ec552-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:21:31.501861 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.501832 7010 generic.go:356] "Generic (PLEG): container finished" podID="2adf6e82-f5f4-48cc-aca3-85a34f5ec552" containerID="6ebc71d3ea9ad9fbde49c8c7c2261aa68cb9ba8948f255306901bbcc1283b863" exitCode=0 Oct 30 13:21:31.501955 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.501876 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" Oct 30 13:21:31.501955 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.501911 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" event={"ID":"2adf6e82-f5f4-48cc-aca3-85a34f5ec552","Type":"ContainerDied","Data":"6ebc71d3ea9ad9fbde49c8c7c2261aa68cb9ba8948f255306901bbcc1283b863"} Oct 30 13:21:31.501955 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.501953 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk" event={"ID":"2adf6e82-f5f4-48cc-aca3-85a34f5ec552","Type":"ContainerDied","Data":"d6c106f0870a1e1e6dfb0d952fffc68e71a901fa9ca3998de966e11f542d9c81"} Oct 30 13:21:31.502130 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.501975 7010 scope.go:122] "RemoveContainer" containerID="6ebc71d3ea9ad9fbde49c8c7c2261aa68cb9ba8948f255306901bbcc1283b863" Oct 30 13:21:31.503640 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.503613 7010 generic.go:356] "Generic (PLEG): container finished" podID="03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" containerID="9af61fdb229f6152c7f3200bded503340a66729535da56c7702ace8ed3f19a3e" exitCode=0 Oct 30 13:21:31.503922 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.503687 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" Oct 30 13:21:31.503922 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.503746 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" event={"ID":"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d","Type":"ContainerDied","Data":"9af61fdb229f6152c7f3200bded503340a66729535da56c7702ace8ed3f19a3e"} Oct 30 13:21:31.503922 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.503788 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl" event={"ID":"03b1816f-01c8-4c93-8ba3-ffdd5a1be50d","Type":"ContainerDied","Data":"f0302edd47f695d02c872331432b80180e5ca4eb48842beb014982cf5e52af05"} Oct 30 13:21:31.512927 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.512910 7010 scope.go:122] "RemoveContainer" containerID="6ebc71d3ea9ad9fbde49c8c7c2261aa68cb9ba8948f255306901bbcc1283b863" Oct 30 13:21:31.513206 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:21:31.513189 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ebc71d3ea9ad9fbde49c8c7c2261aa68cb9ba8948f255306901bbcc1283b863\": container with ID starting with 6ebc71d3ea9ad9fbde49c8c7c2261aa68cb9ba8948f255306901bbcc1283b863 not found: ID does not exist" containerID="6ebc71d3ea9ad9fbde49c8c7c2261aa68cb9ba8948f255306901bbcc1283b863" Oct 30 13:21:31.513270 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.513212 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebc71d3ea9ad9fbde49c8c7c2261aa68cb9ba8948f255306901bbcc1283b863"} err="failed to get container status \"6ebc71d3ea9ad9fbde49c8c7c2261aa68cb9ba8948f255306901bbcc1283b863\": rpc error: code = NotFound desc = could not find container \"6ebc71d3ea9ad9fbde49c8c7c2261aa68cb9ba8948f255306901bbcc1283b863\": container with ID starting with 6ebc71d3ea9ad9fbde49c8c7c2261aa68cb9ba8948f255306901bbcc1283b863 not found: ID does not exist" Oct 30 13:21:31.513270 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.513228 7010 scope.go:122] "RemoveContainer" containerID="9af61fdb229f6152c7f3200bded503340a66729535da56c7702ace8ed3f19a3e" Oct 30 13:21:31.522611 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.522595 7010 scope.go:122] "RemoveContainer" containerID="9af61fdb229f6152c7f3200bded503340a66729535da56c7702ace8ed3f19a3e" Oct 30 13:21:31.522895 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:21:31.522858 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af61fdb229f6152c7f3200bded503340a66729535da56c7702ace8ed3f19a3e\": container with ID starting with 9af61fdb229f6152c7f3200bded503340a66729535da56c7702ace8ed3f19a3e not found: ID does not exist" containerID="9af61fdb229f6152c7f3200bded503340a66729535da56c7702ace8ed3f19a3e" Oct 30 13:21:31.523019 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.522892 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af61fdb229f6152c7f3200bded503340a66729535da56c7702ace8ed3f19a3e"} err="failed to get container status \"9af61fdb229f6152c7f3200bded503340a66729535da56c7702ace8ed3f19a3e\": rpc error: code = NotFound desc = could not find container \"9af61fdb229f6152c7f3200bded503340a66729535da56c7702ace8ed3f19a3e\": container with ID starting with 9af61fdb229f6152c7f3200bded503340a66729535da56c7702ace8ed3f19a3e not found: ID does not exist" Oct 30 13:21:31.558639 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.558614 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk"] Oct 30 13:21:31.571179 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.571154 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-hmvsk"] Oct 30 13:21:31.608688 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.608664 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl"] Oct 30 13:21:31.611361 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:31.611338 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-hnlsl"] Oct 30 13:21:33.492029 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:33.491993 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" path="/var/lib/kubelet/pods/03b1816f-01c8-4c93-8ba3-ffdd5a1be50d/volumes" Oct 30 13:21:33.492745 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:33.492721 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2adf6e82-f5f4-48cc-aca3-85a34f5ec552" path="/var/lib/kubelet/pods/2adf6e82-f5f4-48cc-aca3-85a34f5ec552/volumes" Oct 30 13:21:52.820564 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:52.820529 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk"] Oct 30 13:21:52.821151 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:52.821137 7010 cpu_manager.go:414] "containerMap: removing container" podUID="2adf6e82-f5f4-48cc-aca3-85a34f5ec552" containerName="istio-proxy" Oct 30 13:21:52.821201 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:52.821152 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2adf6e82-f5f4-48cc-aca3-85a34f5ec552" containerName="istio-proxy" Oct 30 13:21:52.821201 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:52.821198 7010 cpu_manager.go:414] "containerMap: removing container" podUID="03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" containerName="istio-proxy" Oct 30 13:21:52.821278 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:52.821205 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" containerName="istio-proxy" Oct 30 13:21:52.821320 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:52.821312 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b1816f-01c8-4c93-8ba3-ffdd5a1be50d" containerName="istio-proxy" Oct 30 13:21:52.821356 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:52.821324 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="2adf6e82-f5f4-48cc-aca3-85a34f5ec552" containerName="istio-proxy" Oct 30 13:21:52.825081 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:52.825063 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" Oct 30 13:21:52.846627 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:52.846603 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk"] Oct 30 13:21:52.855882 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:52.855850 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5c7ba3\"" Oct 30 13:21:52.856051 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:52.856022 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmpq8\" (UniqueName: \"kubernetes.io/projected/7459ef64-d4fa-4130-acd8-4a0162654b1d-kube-api-access-mmpq8\") pod \"payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk\" (UID: \"7459ef64-d4fa-4130-acd8-4a0162654b1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" Oct 30 13:21:52.856167 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:52.856141 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7459ef64-d4fa-4130-acd8-4a0162654b1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk\" (UID: \"7459ef64-d4fa-4130-acd8-4a0162654b1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" Oct 30 13:21:52.958466 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:52.958425 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmpq8\" (UniqueName: \"kubernetes.io/projected/7459ef64-d4fa-4130-acd8-4a0162654b1d-kube-api-access-mmpq8\") pod \"payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk\" (UID: \"7459ef64-d4fa-4130-acd8-4a0162654b1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" Oct 30 13:21:52.958686 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:52.958518 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7459ef64-d4fa-4130-acd8-4a0162654b1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk\" (UID: \"7459ef64-d4fa-4130-acd8-4a0162654b1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" Oct 30 13:21:52.959560 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:52.959482 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7459ef64-d4fa-4130-acd8-4a0162654b1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk\" (UID: \"7459ef64-d4fa-4130-acd8-4a0162654b1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" Oct 30 13:21:52.997335 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:52.997222 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmpq8\" (UniqueName: \"kubernetes.io/projected/7459ef64-d4fa-4130-acd8-4a0162654b1d-kube-api-access-mmpq8\") pod \"payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk\" (UID: \"7459ef64-d4fa-4130-acd8-4a0162654b1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" Oct 30 13:21:53.117581 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.117470 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl"] Oct 30 13:21:53.122126 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.122096 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" Oct 30 13:21:53.146919 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.146886 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5c7ba3-dockercfg-n5xt7\"" Oct 30 13:21:53.147596 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.147574 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" Oct 30 13:21:53.160788 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.160755 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl"] Oct 30 13:21:53.164617 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.164585 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w9cc\" (UniqueName: \"kubernetes.io/projected/353779fe-c229-48e5-b17b-095bd6c32f59-kube-api-access-4w9cc\") pod \"payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl\" (UID: \"353779fe-c229-48e5-b17b-095bd6c32f59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" Oct 30 13:21:53.164780 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.164753 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353779fe-c229-48e5-b17b-095bd6c32f59-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl\" (UID: \"353779fe-c229-48e5-b17b-095bd6c32f59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" Oct 30 13:21:53.280536 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.267265 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w9cc\" (UniqueName: \"kubernetes.io/projected/353779fe-c229-48e5-b17b-095bd6c32f59-kube-api-access-4w9cc\") pod \"payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl\" (UID: \"353779fe-c229-48e5-b17b-095bd6c32f59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" Oct 30 13:21:53.280536 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.267414 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353779fe-c229-48e5-b17b-095bd6c32f59-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl\" (UID: \"353779fe-c229-48e5-b17b-095bd6c32f59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" Oct 30 13:21:53.280536 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.273542 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353779fe-c229-48e5-b17b-095bd6c32f59-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl\" (UID: \"353779fe-c229-48e5-b17b-095bd6c32f59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" Oct 30 13:21:53.304537 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.304467 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w9cc\" (UniqueName: \"kubernetes.io/projected/353779fe-c229-48e5-b17b-095bd6c32f59-kube-api-access-4w9cc\") pod \"payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl\" (UID: \"353779fe-c229-48e5-b17b-095bd6c32f59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" Oct 30 13:21:53.451658 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.451623 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" Oct 30 13:21:53.516407 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.516378 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk"] Oct 30 13:21:53.676898 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.676850 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" event={"ID":"7459ef64-d4fa-4130-acd8-4a0162654b1d","Type":"ContainerStarted","Data":"b7ccb5d3b14ad984fc68fcdb5cf57281abec7877e79bc560ac546d601c38630e"} Oct 30 13:21:53.677068 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.676916 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" event={"ID":"7459ef64-d4fa-4130-acd8-4a0162654b1d","Type":"ContainerStarted","Data":"d8ef1fe062ab5323537588ab20a67f5437d0a9040ffb2ae58e68c3c648f43994"} Oct 30 13:21:53.677068 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.676943 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" Oct 30 13:21:53.721563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.721460 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" podStartSLOduration=1.721437385 podStartE2EDuration="1.721437385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:53.718945299 +0000 UTC m=+1770.351132460" watchObservedRunningTime="2026-10-30 13:21:53.721437385 +0000 UTC m=+1770.353624547" Oct 30 13:21:53.775024 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:53.774996 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl"] Oct 30 13:21:53.775885 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:21:53.775856 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353779fe_c229_48e5_b17b_095bd6c32f59.slice/crio-7cd89c0fee4fcb432c4cd2057f98ed0539f9b2065013de092ebbace104bcab3e WatchSource:0}: Error finding container 7cd89c0fee4fcb432c4cd2057f98ed0539f9b2065013de092ebbace104bcab3e: Status 404 returned error can't find the container with id 7cd89c0fee4fcb432c4cd2057f98ed0539f9b2065013de092ebbace104bcab3e Oct 30 13:21:54.684387 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:54.684347 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" event={"ID":"353779fe-c229-48e5-b17b-095bd6c32f59","Type":"ContainerStarted","Data":"ef437412f0788ce10ba1f38302921da48c4a016f1a808a3bd0925c9b665d2636"} Oct 30 13:21:54.684810 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:54.684412 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" event={"ID":"353779fe-c229-48e5-b17b-095bd6c32f59","Type":"ContainerStarted","Data":"7cd89c0fee4fcb432c4cd2057f98ed0539f9b2065013de092ebbace104bcab3e"} Oct 30 13:21:54.684810 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:54.684646 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" Oct 30 13:21:54.729833 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:21:54.729776 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" podStartSLOduration=1.729758221 podStartE2EDuration="1.729758221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:54.727239699 +0000 UTC m=+1771.359426861" watchObservedRunningTime="2026-10-30 13:21:54.729758221 +0000 UTC m=+1771.361945384" Oct 30 13:21:59.403510 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:21:59.402854 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:22:00.787392 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.787361 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt"] Oct 30 13:22:00.792582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.792554 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:00.814334 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.814281 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8467ed\"/\"openshift-service-ca.crt\"" Oct 30 13:22:00.814334 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.814323 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8467ed\"/\"kube-root-ca.crt\"" Oct 30 13:22:00.820303 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.820275 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt"] Oct 30 13:22:00.829774 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.829744 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8467ed\"/\"test-model-8467ed-kserve-self-signed-certs\"" Oct 30 13:22:00.829774 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.829766 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8467ed\"/\"default-dockercfg-jdmfk\"" Oct 30 13:22:00.890347 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.890311 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97c7r\" (UniqueName: \"kubernetes.io/projected/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-kube-api-access-97c7r\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:00.890588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.890561 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-tmp-dir\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:00.890699 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.890667 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-tls-certs\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:00.890757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.890717 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-dshm\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:00.890811 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.890773 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-model-cache\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:00.890849 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.890822 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-home\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:00.992398 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.992364 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97c7r\" (UniqueName: \"kubernetes.io/projected/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-kube-api-access-97c7r\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:00.992620 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.992525 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-tmp-dir\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:00.992620 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.992596 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-tls-certs\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:00.992756 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.992635 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-dshm\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:00.992756 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.992689 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-model-cache\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:00.993088 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.993067 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-model-cache\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:00.993461 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.993427 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-tmp-dir\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:00.993600 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.993524 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-home\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:00.993954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.993929 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-home\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:00.995950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.995922 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-dshm\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:00.997029 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:00.997007 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-tls-certs\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:01.018201 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:01.018166 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97c7r\" (UniqueName: \"kubernetes.io/projected/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-kube-api-access-97c7r\") pod \"test-model-8467ed-kserve-74bf9d887-stmrt\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:01.110954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:01.110854 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:01.432468 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:01.432431 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt"] Oct 30 13:22:01.435956 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:22:01.435931 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81c361d4_8afa_4d5a_a827_ef4bf03a73c0.slice/crio-2781682f4ae475a2204cb6d6071cf59ee1c24e46115d5ccd01d83d0d0db2e164 WatchSource:0}: Error finding container 2781682f4ae475a2204cb6d6071cf59ee1c24e46115d5ccd01d83d0d0db2e164: Status 404 returned error can't find the container with id 2781682f4ae475a2204cb6d6071cf59ee1c24e46115d5ccd01d83d0d0db2e164 Oct 30 13:22:01.744193 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:01.744097 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" event={"ID":"81c361d4-8afa-4d5a-a827-ef4bf03a73c0","Type":"ContainerStarted","Data":"15c2cfaeae3e26932c5cce7f3674c5d5df05f56d131e73f447a11950374e165c"} Oct 30 13:22:01.744193 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:01.744150 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" event={"ID":"81c361d4-8afa-4d5a-a827-ef4bf03a73c0","Type":"ContainerStarted","Data":"2781682f4ae475a2204cb6d6071cf59ee1c24e46115d5ccd01d83d0d0db2e164"} Oct 30 13:22:01.797507 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:01.792877 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" podStartSLOduration=1.792857033 podStartE2EDuration="1.792857033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:01.790901422 +0000 UTC m=+1778.423088597" watchObservedRunningTime="2026-10-30 13:22:01.792857033 +0000 UTC m=+1778.425044199" Oct 30 13:22:02.111906 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:02.111774 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:04.702514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:04.699674 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" Oct 30 13:22:05.708738 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:05.708690 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" Oct 30 13:22:11.111331 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.111298 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:11.126401 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.126369 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:11.378478 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.378446 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws"] Oct 30 13:22:11.382950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.382930 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" Oct 30 13:22:11.401385 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.401358 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-376c89\"" Oct 30 13:22:11.405231 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.405181 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws"] Oct 30 13:22:11.446640 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.446610 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-376c89-dockercfg-t44fs\"" Oct 30 13:22:11.474793 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.474757 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm"] Oct 30 13:22:11.481735 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.480995 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" Oct 30 13:22:11.536387 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.535925 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef58e88a-0bad-4099-9194-d4cd16997478-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws\" (UID: \"ef58e88a-0bad-4099-9194-d4cd16997478\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" Oct 30 13:22:11.536387 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.536079 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdssr\" (UniqueName: \"kubernetes.io/projected/ef58e88a-0bad-4099-9194-d4cd16997478-kube-api-access-fdssr\") pod \"payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws\" (UID: \"ef58e88a-0bad-4099-9194-d4cd16997478\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" Oct 30 13:22:11.597694 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.597657 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm"] Oct 30 13:22:11.644591 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.637694 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef58e88a-0bad-4099-9194-d4cd16997478-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws\" (UID: \"ef58e88a-0bad-4099-9194-d4cd16997478\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" Oct 30 13:22:11.644591 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.637793 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z46ql\" (UniqueName: \"kubernetes.io/projected/d1c819c5-dff1-4b26-a66f-d4d2f6376387-kube-api-access-z46ql\") pod \"payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm\" (UID: \"d1c819c5-dff1-4b26-a66f-d4d2f6376387\") " pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" Oct 30 13:22:11.644591 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.637931 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdssr\" (UniqueName: \"kubernetes.io/projected/ef58e88a-0bad-4099-9194-d4cd16997478-kube-api-access-fdssr\") pod \"payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws\" (UID: \"ef58e88a-0bad-4099-9194-d4cd16997478\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" Oct 30 13:22:11.644591 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.638050 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c819c5-dff1-4b26-a66f-d4d2f6376387-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm\" (UID: \"d1c819c5-dff1-4b26-a66f-d4d2f6376387\") " pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" Oct 30 13:22:11.644591 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.639106 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef58e88a-0bad-4099-9194-d4cd16997478-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws\" (UID: \"ef58e88a-0bad-4099-9194-d4cd16997478\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" Oct 30 13:22:11.665188 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.665130 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdssr\" (UniqueName: \"kubernetes.io/projected/ef58e88a-0bad-4099-9194-d4cd16997478-kube-api-access-fdssr\") pod \"payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws\" (UID: \"ef58e88a-0bad-4099-9194-d4cd16997478\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" Oct 30 13:22:11.698357 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.698327 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" Oct 30 13:22:11.739840 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.739798 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46ql\" (UniqueName: \"kubernetes.io/projected/d1c819c5-dff1-4b26-a66f-d4d2f6376387-kube-api-access-z46ql\") pod \"payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm\" (UID: \"d1c819c5-dff1-4b26-a66f-d4d2f6376387\") " pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" Oct 30 13:22:11.740039 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.740017 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c819c5-dff1-4b26-a66f-d4d2f6376387-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm\" (UID: \"d1c819c5-dff1-4b26-a66f-d4d2f6376387\") " pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" Oct 30 13:22:11.741188 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.741134 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c819c5-dff1-4b26-a66f-d4d2f6376387-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm\" (UID: \"d1c819c5-dff1-4b26-a66f-d4d2f6376387\") " pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" Oct 30 13:22:11.768623 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.767088 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46ql\" (UniqueName: \"kubernetes.io/projected/d1c819c5-dff1-4b26-a66f-d4d2f6376387-kube-api-access-z46ql\") pod \"payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm\" (UID: \"d1c819c5-dff1-4b26-a66f-d4d2f6376387\") " pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" Oct 30 13:22:11.872936 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.872899 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:22:11.887928 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:11.887896 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" Oct 30 13:22:12.035715 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:12.035688 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws"] Oct 30 13:22:12.234249 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:12.234224 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm"] Oct 30 13:22:12.236221 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:22:12.236198 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c819c5_dff1_4b26_a66f_d4d2f6376387.slice/crio-6e296f3146a62a7c821ada4556cc249027ec825dc6e0f55bff890fe9f9bd2ad0 WatchSource:0}: Error finding container 6e296f3146a62a7c821ada4556cc249027ec825dc6e0f55bff890fe9f9bd2ad0: Status 404 returned error can't find the container with id 6e296f3146a62a7c821ada4556cc249027ec825dc6e0f55bff890fe9f9bd2ad0 Oct 30 13:22:12.867196 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:12.867160 7010 generic.go:356] "Generic (PLEG): container finished" podID="51645cac-ccad-44c3-9928-d0283dc273ee" containerID="203d983a17ecd15c8e8f0f201bc685f37df111f4713ea8702c50ec0cd7e3cfc9" exitCode=0 Oct 30 13:22:12.867397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:12.867245 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" event={"ID":"51645cac-ccad-44c3-9928-d0283dc273ee","Type":"ContainerDied","Data":"203d983a17ecd15c8e8f0f201bc685f37df111f4713ea8702c50ec0cd7e3cfc9"} Oct 30 13:22:12.867721 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:12.867695 7010 scope.go:122] "RemoveContainer" containerID="203d983a17ecd15c8e8f0f201bc685f37df111f4713ea8702c50ec0cd7e3cfc9" Oct 30 13:22:12.869332 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:12.869310 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" event={"ID":"d1c819c5-dff1-4b26-a66f-d4d2f6376387","Type":"ContainerStarted","Data":"6d7e784beb25ef5803d3587e7e3992460f6357cbaae9f0085f8129152a489a1a"} Oct 30 13:22:12.869464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:12.869364 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" Oct 30 13:22:12.869464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:12.869392 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" event={"ID":"d1c819c5-dff1-4b26-a66f-d4d2f6376387","Type":"ContainerStarted","Data":"6e296f3146a62a7c821ada4556cc249027ec825dc6e0f55bff890fe9f9bd2ad0"} Oct 30 13:22:12.869896 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:12.869860 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:22:12.869998 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:12.869942 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:22:12.870063 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:12.870009 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:22:12.872030 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:12.872007 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" event={"ID":"ef58e88a-0bad-4099-9194-d4cd16997478","Type":"ContainerStarted","Data":"460db5f17ee38a8decce8c834e64196aac68f83abb4881848d4ab63c4361dba4"} Oct 30 13:22:12.872141 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:12.872047 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" event={"ID":"ef58e88a-0bad-4099-9194-d4cd16997478","Type":"ContainerStarted","Data":"99713d981dd913ccbfc89d070860beeb4bfb1b85d33fa837ba55d4c7588d5fa2"} Oct 30 13:22:12.890859 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:12.890841 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:22:13.077330 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:13.077272 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" podStartSLOduration=2.077258502 podStartE2EDuration="2.077258502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:13.075562499 +0000 UTC m=+1789.707749662" watchObservedRunningTime="2026-10-30 13:22:13.077258502 +0000 UTC m=+1789.709445669" Oct 30 13:22:13.126437 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:13.126379 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" podStartSLOduration=2.126360499 podStartE2EDuration="2.126360499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:13.125932013 +0000 UTC m=+1789.758119176" watchObservedRunningTime="2026-10-30 13:22:13.126360499 +0000 UTC m=+1789.758547666" Oct 30 13:22:13.887591 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:13.887552 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" event={"ID":"51645cac-ccad-44c3-9928-d0283dc273ee","Type":"ContainerStarted","Data":"c3724b123a8783f7c38e26175f991dfe5786b82f0f2c5443a05a1e3d7166489b"} Oct 30 13:22:13.888225 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:13.888201 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" Oct 30 13:22:14.112511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:14.110228 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:22:14.112511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:14.110360 7010 patch_prober.go:29] interesting pod/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:22:14.112511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:14.110404 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" podUID="51645cac-ccad-44c3-9928-d0283dc273ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 13:22:15.108927 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:15.108887 7010 patch_prober.go:29] interesting pod/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:22:15.109321 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:15.108948 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" podUID="51645cac-ccad-44c3-9928-d0283dc273ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 13:22:16.030777 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.030736 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb"] Oct 30 13:22:16.035753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.035728 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.069104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.069074 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb"] Oct 30 13:22:16.115555 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.112976 7010 patch_prober.go:29] interesting pod/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:22:16.115555 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.113049 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" podUID="51645cac-ccad-44c3-9928-d0283dc273ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 13:22:16.133201 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.133169 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.133368 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.133331 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.133455 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.133385 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.133542 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.133521 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.133606 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.133569 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.133688 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.133666 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36015964-d36a-428a-8c47-367689174d12-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.133818 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.133756 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zmnr\" (UniqueName: \"kubernetes.io/projected/36015964-d36a-428a-8c47-367689174d12-kube-api-access-9zmnr\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.133899 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.133881 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36015964-d36a-428a-8c47-367689174d12-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.134001 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.133985 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36015964-d36a-428a-8c47-367689174d12-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.140338 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.140315 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-8d82d5-openshift-default-dockercfg-w5n4b\"" Oct 30 13:22:16.234949 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.234914 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36015964-d36a-428a-8c47-367689174d12-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.235115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.235000 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36015964-d36a-428a-8c47-367689174d12-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.235115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.235044 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.235193 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.235120 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.235193 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.235158 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.235304 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.235229 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.235304 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.235268 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.235404 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.235302 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36015964-d36a-428a-8c47-367689174d12-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.235404 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.235357 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zmnr\" (UniqueName: \"kubernetes.io/projected/36015964-d36a-428a-8c47-367689174d12-kube-api-access-9zmnr\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.235995 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.235936 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.236198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.236167 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36015964-d36a-428a-8c47-367689174d12-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.240178 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.236782 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.240178 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.238116 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36015964-d36a-428a-8c47-367689174d12-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.240178 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.238248 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.240178 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.238293 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.249005 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.248978 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.260105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.260082 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36015964-d36a-428a-8c47-367689174d12-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.260291 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.260271 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zmnr\" (UniqueName: \"kubernetes.io/projected/36015964-d36a-428a-8c47-367689174d12-kube-api-access-9zmnr\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.351073 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.350995 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:16.734850 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.734811 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb"] Oct 30 13:22:16.736278 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:22:16.736249 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36015964_d36a_428a_8c47_367689174d12.slice/crio-ef02862102e1d880183748b2442e14d4a51508d27c2185f11a35f4c71c79648f WatchSource:0}: Error finding container ef02862102e1d880183748b2442e14d4a51508d27c2185f11a35f4c71c79648f: Status 404 returned error can't find the container with id ef02862102e1d880183748b2442e14d4a51508d27c2185f11a35f4c71c79648f Oct 30 13:22:16.739409 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.739373 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:22:16.739547 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.739440 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:22:16.739547 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.739469 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:22:16.932704 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.932650 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" event={"ID":"36015964-d36a-428a-8c47-367689174d12","Type":"ContainerStarted","Data":"a03998313cc8485ece3331d041d6bd2a13aa7913114d68ec83ec815d894652bc"} Oct 30 13:22:16.932889 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.932719 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" event={"ID":"36015964-d36a-428a-8c47-367689174d12","Type":"ContainerStarted","Data":"ef02862102e1d880183748b2442e14d4a51508d27c2185f11a35f4c71c79648f"} Oct 30 13:22:16.982923 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:16.982814 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" podStartSLOduration=0.982795521 podStartE2EDuration="982.795521ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:16.978839293 +0000 UTC m=+1793.611026456" watchObservedRunningTime="2026-10-30 13:22:16.982795521 +0000 UTC m=+1793.614982683" Oct 30 13:22:17.111511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:17.110637 7010 patch_prober.go:29] interesting pod/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:22:17.111511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:17.110729 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" podUID="51645cac-ccad-44c3-9928-d0283dc273ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 13:22:17.351779 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:17.351525 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:17.353110 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:17.353079 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 13:22:17.353230 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:17.353132 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" podUID="36015964-d36a-428a-8c47-367689174d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 13:22:18.109805 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:18.109632 7010 patch_prober.go:29] interesting pod/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:22:18.109805 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:18.109691 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" podUID="51645cac-ccad-44c3-9928-d0283dc273ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 13:22:18.354590 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:18.352600 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 13:22:18.354590 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:18.352665 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" podUID="36015964-d36a-428a-8c47-367689174d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 13:22:19.119938 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:19.119898 7010 patch_prober.go:29] interesting pod/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 13:22:19.120228 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:19.120198 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" podUID="51645cac-ccad-44c3-9928-d0283dc273ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 13:22:19.351638 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:19.351591 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 13:22:19.351818 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:19.351669 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" podUID="36015964-d36a-428a-8c47-367689174d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 13:22:20.114655 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:20.114623 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:22:20.115093 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:20.114934 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:22:20.115714 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:20.115695 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:22:20.356525 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:20.356440 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 13:22:20.356723 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:20.356523 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" podUID="36015964-d36a-428a-8c47-367689174d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 13:22:21.351845 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:21.351809 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 13:22:21.352272 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:21.351879 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" podUID="36015964-d36a-428a-8c47-367689174d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 13:22:22.351972 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:22.351936 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 13:22:22.352549 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:22.352005 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" podUID="36015964-d36a-428a-8c47-367689174d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 13:22:23.351735 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:23.351699 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 13:22:23.351910 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:23.351757 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" podUID="36015964-d36a-428a-8c47-367689174d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 13:22:23.785217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:23.785183 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:22:23.787222 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:23.787200 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:22:23.892151 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:23.892122 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" Oct 30 13:22:24.356915 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:24.356883 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:24.357315 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:24.357297 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:24.358041 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:24.358020 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:22:24.915224 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:24.915192 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" Oct 30 13:22:29.401793 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:22:29.401753 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:22:45.937399 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:45.937321 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d"] Oct 30 13:22:45.941634 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:45.941612 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:45.964732 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:45.964691 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d"] Oct 30 13:22:45.967746 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:45.967719 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-home\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:45.967896 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:45.967769 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-dshm\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:45.967966 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:45.967900 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-tls-certs\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:45.968022 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:45.967967 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-tmp-dir\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:45.968115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:45.968092 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78sd7\" (UniqueName: \"kubernetes.io/projected/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-kube-api-access-78sd7\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:45.968339 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:45.968323 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-model-cache\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:45.969732 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:45.969709 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5dd8cc\"/\"test-model-5dd8cc-kserve-self-signed-certs\"" Oct 30 13:22:45.973677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:45.973660 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5dd8cc\"/\"default-dockercfg-lsbsq\"" Oct 30 13:22:45.973775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:45.973749 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5dd8cc\"/\"openshift-service-ca.crt\"" Oct 30 13:22:45.973827 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:45.973774 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5dd8cc\"/\"kube-root-ca.crt\"" Oct 30 13:22:46.069409 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:46.069373 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-tls-certs\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:46.069624 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:46.069446 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-tmp-dir\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:46.069624 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:46.069527 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78sd7\" (UniqueName: \"kubernetes.io/projected/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-kube-api-access-78sd7\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:46.069809 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:46.069791 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-model-cache\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:46.069888 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:46.069857 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-home\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:46.069959 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:46.069903 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-dshm\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:46.070019 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:46.069959 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-tmp-dir\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:46.070162 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:46.070141 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-model-cache\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:46.070258 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:46.070238 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-home\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:46.072249 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:46.072224 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-dshm\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:46.072834 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:46.072808 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-tls-certs\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:46.111869 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:46.111834 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78sd7\" (UniqueName: \"kubernetes.io/projected/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-kube-api-access-78sd7\") pod \"test-model-5dd8cc-kserve-64fb5df7df-mkw6d\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:46.261522 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:46.261411 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:46.672055 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:46.672019 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d"] Oct 30 13:22:46.673158 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:22:46.673130 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b9c833b_eab4_42d4_bda7_fac3c9c7455c.slice/crio-1fe5b9ef1c457aa547788cdffa5b8cc75dfe0d4b9b1a32daf406516912f3fec0 WatchSource:0}: Error finding container 1fe5b9ef1c457aa547788cdffa5b8cc75dfe0d4b9b1a32daf406516912f3fec0: Status 404 returned error can't find the container with id 1fe5b9ef1c457aa547788cdffa5b8cc75dfe0d4b9b1a32daf406516912f3fec0 Oct 30 13:22:47.265061 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:47.265014 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" event={"ID":"6b9c833b-eab4-42d4-bda7-fac3c9c7455c","Type":"ContainerStarted","Data":"9fdc414eb2516b9772fc0e96b8b057c02f20ad8f19c323f52c649c789d5c3e1a"} Oct 30 13:22:47.265557 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:47.265070 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" event={"ID":"6b9c833b-eab4-42d4-bda7-fac3c9c7455c","Type":"ContainerStarted","Data":"1fe5b9ef1c457aa547788cdffa5b8cc75dfe0d4b9b1a32daf406516912f3fec0"} Oct 30 13:22:47.320402 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:47.320337 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" podStartSLOduration=2.320315905 podStartE2EDuration="2.320315905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:47.31525107 +0000 UTC m=+1823.947438232" watchObservedRunningTime="2026-10-30 13:22:47.320315905 +0000 UTC m=+1823.952503072" Oct 30 13:22:48.262632 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:48.262593 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:50.889543 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:50.877759 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2"] Oct 30 13:22:50.889543 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:50.882792 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:50.912465 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:50.912415 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-ff4285\"/\"openshift-service-ca.crt\"" Oct 30 13:22:50.912696 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:50.912499 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-ff4285\"/\"kube-root-ca.crt\"" Oct 30 13:22:50.912696 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:50.912612 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-ff4285\"/\"auth-test-model-ff4285-kserve-self-signed-certs\"" Oct 30 13:22:50.915507 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:50.915462 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-ff4285\"/\"default-dockercfg-565c8\"" Oct 30 13:22:50.919205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:50.919177 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2"] Oct 30 13:22:50.954497 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:50.954466 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-model-cache\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:50.954655 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:50.954551 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-tmp-dir\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:50.954706 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:50.954672 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-dshm\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:50.954967 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:50.954947 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe53a3fb-1f3d-463b-a165-713c55449df7-tls-certs\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:50.955014 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:50.954988 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-home\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:50.955100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:50.955085 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmpmb\" (UniqueName: \"kubernetes.io/projected/fe53a3fb-1f3d-463b-a165-713c55449df7-kube-api-access-dmpmb\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:51.056566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:51.056531 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe53a3fb-1f3d-463b-a165-713c55449df7-tls-certs\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:51.056566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:51.056572 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-home\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:51.056835 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:51.056618 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmpmb\" (UniqueName: \"kubernetes.io/projected/fe53a3fb-1f3d-463b-a165-713c55449df7-kube-api-access-dmpmb\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:51.056835 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:51.056696 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-model-cache\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:51.056835 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:51.056738 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-tmp-dir\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:51.056835 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:51.056792 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-dshm\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:51.057076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:51.057049 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-home\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:51.057364 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:51.057328 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-model-cache\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:51.057462 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:51.057358 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-tmp-dir\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:51.059150 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:51.059123 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-dshm\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:51.059367 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:51.059347 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe53a3fb-1f3d-463b-a165-713c55449df7-tls-certs\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:51.089231 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:51.089193 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmpmb\" (UniqueName: \"kubernetes.io/projected/fe53a3fb-1f3d-463b-a165-713c55449df7-kube-api-access-dmpmb\") pod \"auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:51.199595 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:51.198936 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:51.675272 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:51.675240 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2"] Oct 30 13:22:51.676535 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:22:51.676511 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe53a3fb_1f3d_463b_a165_713c55449df7.slice/crio-693807bd4c24d180dc95c07560f18321d725bbca4bd687e70f101f33fdb4e0b4 WatchSource:0}: Error finding container 693807bd4c24d180dc95c07560f18321d725bbca4bd687e70f101f33fdb4e0b4: Status 404 returned error can't find the container with id 693807bd4c24d180dc95c07560f18321d725bbca4bd687e70f101f33fdb4e0b4 Oct 30 13:22:52.356762 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:52.356722 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" event={"ID":"fe53a3fb-1f3d-463b-a165-713c55449df7","Type":"ContainerStarted","Data":"20ba82d1fb789aaa2ab3103fa870f97aad1b1a74473cdb882726111cdcd23bc6"} Oct 30 13:22:52.356762 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:52.356769 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" event={"ID":"fe53a3fb-1f3d-463b-a165-713c55449df7","Type":"ContainerStarted","Data":"693807bd4c24d180dc95c07560f18321d725bbca4bd687e70f101f33fdb4e0b4"} Oct 30 13:22:52.401721 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:52.401667 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" podStartSLOduration=2.401653924 podStartE2EDuration="2.401653924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:52.399015255 +0000 UTC m=+1829.031202416" watchObservedRunningTime="2026-10-30 13:22:52.401653924 +0000 UTC m=+1829.033841087" Oct 30 13:22:53.200292 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:53.200256 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:22:56.261397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:56.261357 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:56.274392 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:56.274366 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:56.397006 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:22:56.396975 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:22:59.403986 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:22:59.403940 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:23:01.200056 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:23:01.200010 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:23:01.213777 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:23:01.213747 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:23:01.445074 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:23:01.445037 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:23:29.406580 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:23:29.406537 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:23:59.404645 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:23:59.404602 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:24:11.478723 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.478683 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g"] Oct 30 13:24:11.484213 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.484183 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.505474 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.505439 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g"] Oct 30 13:24:11.509105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.509071 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-home\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.509279 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.509143 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w868v\" (UniqueName: \"kubernetes.io/projected/8422efa7-cca3-4534-983e-108c338e1c16-kube-api-access-w868v\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.509279 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.509179 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-model-cache\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.509279 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.509203 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8422efa7-cca3-4534-983e-108c338e1c16-tls-certs\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.509279 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.509248 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-tmp-dir\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.509524 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.509308 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-dshm\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.510243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.510218 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-47628d\"/\"kube-root-ca.crt\"" Oct 30 13:24:11.510243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.510218 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-47628d\"/\"openshift-service-ca.crt\"" Oct 30 13:24:11.510770 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.510748 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-47628d\"/\"test-model-47628d-kserve-self-signed-certs\"" Oct 30 13:24:11.513452 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.513412 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-47628d\"/\"default-dockercfg-6d4cg\"" Oct 30 13:24:11.611172 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.611132 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-dshm\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.611358 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.611298 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-home\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.611423 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.611378 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w868v\" (UniqueName: \"kubernetes.io/projected/8422efa7-cca3-4534-983e-108c338e1c16-kube-api-access-w868v\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.611519 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.611430 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-model-cache\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.611519 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.611468 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8422efa7-cca3-4534-983e-108c338e1c16-tls-certs\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.611834 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.611798 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-home\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.611967 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.611832 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-model-cache\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.612044 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.611998 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-tmp-dir\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.612087 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.611600 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-tmp-dir\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.614628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.614602 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8422efa7-cca3-4534-983e-108c338e1c16-tls-certs\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.614858 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.614838 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-dshm\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.637334 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.637295 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w868v\" (UniqueName: \"kubernetes.io/projected/8422efa7-cca3-4534-983e-108c338e1c16-kube-api-access-w868v\") pod \"test-model-47628d-kserve-765979b8c7-mrn4g\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:11.809694 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:11.809597 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:12.238775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:12.238738 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g"] Oct 30 13:24:12.243740 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:24:12.243691 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8422efa7_cca3_4534_983e_108c338e1c16.slice/crio-02da4cf19d7e62c96eef20d24eea72187d3e99df68101459ee61f3e789634d0b WatchSource:0}: Error finding container 02da4cf19d7e62c96eef20d24eea72187d3e99df68101459ee61f3e789634d0b: Status 404 returned error can't find the container with id 02da4cf19d7e62c96eef20d24eea72187d3e99df68101459ee61f3e789634d0b Oct 30 13:24:13.018180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:13.018137 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" event={"ID":"8422efa7-cca3-4534-983e-108c338e1c16","Type":"ContainerStarted","Data":"3bd9a7ff159760a7a583db1de24f0ea167fb288e4dc223e47ea5fd566eb88fd5"} Oct 30 13:24:13.018733 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:13.018190 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" event={"ID":"8422efa7-cca3-4534-983e-108c338e1c16","Type":"ContainerStarted","Data":"02da4cf19d7e62c96eef20d24eea72187d3e99df68101459ee61f3e789634d0b"} Oct 30 13:24:13.810256 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:13.810212 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:17.231515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.224959 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" podStartSLOduration=6.224934099 podStartE2EDuration="6.224934099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:13.082991155 +0000 UTC m=+1909.715178316" watchObservedRunningTime="2026-10-30 13:24:17.224934099 +0000 UTC m=+1913.857121263" Oct 30 13:24:17.231515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.227637 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz"] Oct 30 13:24:17.238511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.233731 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.287513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.282005 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5c7ba3\"/\"kube-root-ca.crt\"" Oct 30 13:24:17.287513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.282131 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5c7ba3\"/\"openshift-service-ca.crt\"" Oct 30 13:24:17.348514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.334157 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5c7ba3\"/\"test-model-5c7ba3-kserve-self-signed-certs\"" Oct 30 13:24:17.348514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.339549 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5c7ba3\"/\"default-dockercfg-27jfk\"" Oct 30 13:24:17.393882 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.393833 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-model-cache\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.394242 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.394215 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-dshm\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.394653 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.394629 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-275sx\" (UniqueName: \"kubernetes.io/projected/3428a596-3e50-418c-9442-9ee08eb312e1-kube-api-access-275sx\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.396654 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.396628 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-tmp-dir\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.396876 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.396857 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-home\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.397148 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.397126 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3428a596-3e50-418c-9442-9ee08eb312e1-tls-certs\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.401380 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.401343 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz"] Oct 30 13:24:17.508573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.507739 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-model-cache\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.508573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.507814 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-dshm\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.508573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.507910 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-275sx\" (UniqueName: \"kubernetes.io/projected/3428a596-3e50-418c-9442-9ee08eb312e1-kube-api-access-275sx\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.508573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.507983 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-tmp-dir\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.508573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.508009 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-home\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.508573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.508086 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3428a596-3e50-418c-9442-9ee08eb312e1-tls-certs\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.522474 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.516028 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-model-cache\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.522474 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.516718 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-tmp-dir\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.522474 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.517133 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-home\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.545636 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.529121 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3428a596-3e50-418c-9442-9ee08eb312e1-tls-certs\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.563328 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.563278 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-275sx\" (UniqueName: \"kubernetes.io/projected/3428a596-3e50-418c-9442-9ee08eb312e1-kube-api-access-275sx\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.621307 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.621270 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-dshm\") pod \"test-model-5c7ba3-kserve-5c576d9f49-hr4wz\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:17.706236 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:17.706195 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:18.330873 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:18.330839 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz"] Oct 30 13:24:18.332858 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:24:18.332821 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3428a596_3e50_418c_9442_9ee08eb312e1.slice/crio-7d658808dbcde7eaa5ff1bdcbb73d049a61372c564029eea2a23b7b1a3041c45 WatchSource:0}: Error finding container 7d658808dbcde7eaa5ff1bdcbb73d049a61372c564029eea2a23b7b1a3041c45: Status 404 returned error can't find the container with id 7d658808dbcde7eaa5ff1bdcbb73d049a61372c564029eea2a23b7b1a3041c45 Oct 30 13:24:19.074859 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:19.074800 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" event={"ID":"3428a596-3e50-418c-9442-9ee08eb312e1","Type":"ContainerStarted","Data":"3e8c66f33d2bdb43cf1e5eed13d928ecdc1e2dfb5cd8ed11aab287f26e4b56d4"} Oct 30 13:24:19.074859 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:19.074864 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" event={"ID":"3428a596-3e50-418c-9442-9ee08eb312e1","Type":"ContainerStarted","Data":"7d658808dbcde7eaa5ff1bdcbb73d049a61372c564029eea2a23b7b1a3041c45"} Oct 30 13:24:19.121866 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:19.121795 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" podStartSLOduration=2.121778577 podStartE2EDuration="2.121778577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:19.118463374 +0000 UTC m=+1915.750650537" watchObservedRunningTime="2026-10-30 13:24:19.121778577 +0000 UTC m=+1915.753965811" Oct 30 13:24:19.708017 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:19.707971 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:21.810960 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:21.810925 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:21.826062 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:21.826034 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:22.110636 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:22.110541 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:24:27.707839 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:27.707782 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:27.720773 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:27.720745 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:28.155731 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:28.155704 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:24:29.404773 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:24:29.404735 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:24:49.495819 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:24:49.495785 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg"] Oct 30 13:24:59.410123 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:24:59.410078 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:25:04.521376 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:04.521328 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" podUID="96f80372-7a4d-464d-a85b-ad4fa0742324" containerName="main" containerID="cri-o://c41c42b1a2a1290999488e99999ac7653932761fda6cb1d94d144057dcc68cec" gracePeriod=45 Oct 30 13:25:04.820074 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:04.820040 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:25:04.948455 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:04.948414 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96f80372-7a4d-464d-a85b-ad4fa0742324-tls-certs\" (UniqueName: \"kubernetes.io/secret/96f80372-7a4d-464d-a85b-ad4fa0742324-tls-certs\") pod \"96f80372-7a4d-464d-a85b-ad4fa0742324\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " Oct 30 13:25:04.948669 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:04.948633 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-model-cache\") pod \"96f80372-7a4d-464d-a85b-ad4fa0742324\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " Oct 30 13:25:04.948737 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:04.948676 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-home\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-home\") pod \"96f80372-7a4d-464d-a85b-ad4fa0742324\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " Oct 30 13:25:04.948855 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:04.948837 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-tmp-dir\") pod \"96f80372-7a4d-464d-a85b-ad4fa0742324\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " Oct 30 13:25:04.948918 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:04.948900 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96f80372-7a4d-464d-a85b-ad4fa0742324-kube-api-access-cth9g\" (UniqueName: \"kubernetes.io/projected/96f80372-7a4d-464d-a85b-ad4fa0742324-kube-api-access-cth9g\") pod \"96f80372-7a4d-464d-a85b-ad4fa0742324\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " Oct 30 13:25:04.948975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:04.948930 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-dshm\") pod \"96f80372-7a4d-464d-a85b-ad4fa0742324\" (UID: \"96f80372-7a4d-464d-a85b-ad4fa0742324\") " Oct 30 13:25:04.949029 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:04.948936 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-model-cache" pod "96f80372-7a4d-464d-a85b-ad4fa0742324" (UID: "96f80372-7a4d-464d-a85b-ad4fa0742324"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:04.949029 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:04.948996 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-home" pod "96f80372-7a4d-464d-a85b-ad4fa0742324" (UID: "96f80372-7a4d-464d-a85b-ad4fa0742324"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:04.949791 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:04.949761 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:04.949903 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:04.949798 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:04.949903 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:04.949773 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-tmp-dir" pod "96f80372-7a4d-464d-a85b-ad4fa0742324" (UID: "96f80372-7a4d-464d-a85b-ad4fa0742324"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:04.951124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:04.951099 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96f80372-7a4d-464d-a85b-ad4fa0742324-tls-certs" pod "96f80372-7a4d-464d-a85b-ad4fa0742324" (UID: "96f80372-7a4d-464d-a85b-ad4fa0742324"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:25:04.951221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:04.951139 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f80372-7a4d-464d-a85b-ad4fa0742324-kube-api-access-cth9g" pod "96f80372-7a4d-464d-a85b-ad4fa0742324" (UID: "96f80372-7a4d-464d-a85b-ad4fa0742324"). InnerVolumeSpecName "kube-api-access-cth9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:04.951221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:04.951187 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-dshm" pod "96f80372-7a4d-464d-a85b-ad4fa0742324" (UID: "96f80372-7a4d-464d-a85b-ad4fa0742324"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:05.051835 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:05.051746 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:05.051835 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:05.051778 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cth9g\" (UniqueName: \"kubernetes.io/projected/96f80372-7a4d-464d-a85b-ad4fa0742324-kube-api-access-cth9g\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:05.051835 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:05.051787 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96f80372-7a4d-464d-a85b-ad4fa0742324-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:05.051835 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:05.051795 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96f80372-7a4d-464d-a85b-ad4fa0742324-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:05.450556 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:05.450479 7010 generic.go:356] "Generic (PLEG): container finished" podID="96f80372-7a4d-464d-a85b-ad4fa0742324" containerID="c41c42b1a2a1290999488e99999ac7653932761fda6cb1d94d144057dcc68cec" exitCode=0 Oct 30 13:25:05.450556 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:05.450543 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" event={"ID":"96f80372-7a4d-464d-a85b-ad4fa0742324","Type":"ContainerDied","Data":"c41c42b1a2a1290999488e99999ac7653932761fda6cb1d94d144057dcc68cec"} Oct 30 13:25:05.450782 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:05.450572 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" Oct 30 13:25:05.450782 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:05.450596 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg" event={"ID":"96f80372-7a4d-464d-a85b-ad4fa0742324","Type":"ContainerDied","Data":"01d7d6b8a5639288a2604da9348f65d69f1fdcf206cc8d5a6bcb04a71ee54748"} Oct 30 13:25:05.450782 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:05.450615 7010 scope.go:122] "RemoveContainer" containerID="c41c42b1a2a1290999488e99999ac7653932761fda6cb1d94d144057dcc68cec" Oct 30 13:25:05.462079 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:05.462058 7010 scope.go:122] "RemoveContainer" containerID="c41c42b1a2a1290999488e99999ac7653932761fda6cb1d94d144057dcc68cec" Oct 30 13:25:05.462365 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:25:05.462348 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c41c42b1a2a1290999488e99999ac7653932761fda6cb1d94d144057dcc68cec\": container with ID starting with c41c42b1a2a1290999488e99999ac7653932761fda6cb1d94d144057dcc68cec not found: ID does not exist" containerID="c41c42b1a2a1290999488e99999ac7653932761fda6cb1d94d144057dcc68cec" Oct 30 13:25:05.462413 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:05.462374 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41c42b1a2a1290999488e99999ac7653932761fda6cb1d94d144057dcc68cec"} err="failed to get container status \"c41c42b1a2a1290999488e99999ac7653932761fda6cb1d94d144057dcc68cec\": rpc error: code = NotFound desc = could not find container \"c41c42b1a2a1290999488e99999ac7653932761fda6cb1d94d144057dcc68cec\": container with ID starting with c41c42b1a2a1290999488e99999ac7653932761fda6cb1d94d144057dcc68cec not found: ID does not exist" Oct 30 13:25:05.506602 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:05.506564 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg"] Oct 30 13:25:05.518784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:05.518747 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ddb271/test-model-ddb271-kserve-56f49859cf-dzxlg"] Oct 30 13:25:07.495190 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:07.495146 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96f80372-7a4d-464d-a85b-ad4fa0742324" path="/var/lib/kubelet/pods/96f80372-7a4d-464d-a85b-ad4fa0742324/volumes" Oct 30 13:25:26.109552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:26.109484 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg"] Oct 30 13:25:26.110148 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:26.109730 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" podUID="239e461e-6255-4e2e-bc8a-b6c62c013c04" containerName="istio-proxy" containerID="cri-o://b01c853ba4b158563b65f2a24374db2916364468d17630d3f8a516e286196e2a" gracePeriod=30 Oct 30 13:25:26.493559 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:26.493520 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4"] Oct 30 13:25:26.493835 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:26.493792 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" podUID="6ea67e42-fdce-4f95-9b23-a62f6fab20dd" containerName="istio-proxy" containerID="cri-o://05c696ae8391022118f109b802d233fa8411ee26b30ad36afe0c5890f1846448" gracePeriod=30 Oct 30 13:25:27.210976 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:27.210933 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6"] Oct 30 13:25:27.211468 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:27.211198 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" podUID="ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" containerName="istio-proxy" containerID="cri-o://9e3ee119c33c324fa4a1a97fd09e6ed29cedaa2c02f76649e64162494bf2498c" gracePeriod=30 Oct 30 13:25:27.339965 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:27.339925 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw"] Oct 30 13:25:29.422255 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:25:29.422214 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:25:31.458650 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.458624 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:25:31.530116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.529105 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-workload-certs\") pod \"239e461e-6255-4e2e-bc8a-b6c62c013c04\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " Oct 30 13:25:31.530116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.529277 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-envoy\") pod \"239e461e-6255-4e2e-bc8a-b6c62c013c04\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " Oct 30 13:25:31.530116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.529317 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-token\" (UniqueName: \"kubernetes.io/projected/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-token\") pod \"239e461e-6255-4e2e-bc8a-b6c62c013c04\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " Oct 30 13:25:31.530116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.529369 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-workload-certs" pod "239e461e-6255-4e2e-bc8a-b6c62c013c04" (UID: "239e461e-6255-4e2e-bc8a-b6c62c013c04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:31.530116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.529399 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/239e461e-6255-4e2e-bc8a-b6c62c013c04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239e461e-6255-4e2e-bc8a-b6c62c013c04-istiod-ca-cert\") pod \"239e461e-6255-4e2e-bc8a-b6c62c013c04\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " Oct 30 13:25:31.530116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.529431 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-data\") pod \"239e461e-6255-4e2e-bc8a-b6c62c013c04\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " Oct 30 13:25:31.530116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.529457 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-credential-socket\") pod \"239e461e-6255-4e2e-bc8a-b6c62c013c04\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " Oct 30 13:25:31.530116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.529566 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239e461e-6255-4e2e-bc8a-b6c62c013c04-kube-api-access-s6lp5\" (UniqueName: \"kubernetes.io/projected/239e461e-6255-4e2e-bc8a-b6c62c013c04-kube-api-access-s6lp5\") pod \"239e461e-6255-4e2e-bc8a-b6c62c013c04\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " Oct 30 13:25:31.530116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.529610 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-podinfo\") pod \"239e461e-6255-4e2e-bc8a-b6c62c013c04\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " Oct 30 13:25:31.530116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.529634 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-workload-socket\") pod \"239e461e-6255-4e2e-bc8a-b6c62c013c04\" (UID: \"239e461e-6255-4e2e-bc8a-b6c62c013c04\") " Oct 30 13:25:31.530116 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.529949 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-data" pod "239e461e-6255-4e2e-bc8a-b6c62c013c04" (UID: "239e461e-6255-4e2e-bc8a-b6c62c013c04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:31.530770 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.530369 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:31.530770 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.530392 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:31.530920 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.530897 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-workload-socket" pod "239e461e-6255-4e2e-bc8a-b6c62c013c04" (UID: "239e461e-6255-4e2e-bc8a-b6c62c013c04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:31.531117 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.531100 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-credential-socket" pod "239e461e-6255-4e2e-bc8a-b6c62c013c04" (UID: "239e461e-6255-4e2e-bc8a-b6c62c013c04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:31.532251 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.532225 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/239e461e-6255-4e2e-bc8a-b6c62c013c04-istiod-ca-cert" pod "239e461e-6255-4e2e-bc8a-b6c62c013c04" (UID: "239e461e-6255-4e2e-bc8a-b6c62c013c04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:31.532539 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.532513 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-envoy" pod "239e461e-6255-4e2e-bc8a-b6c62c013c04" (UID: "239e461e-6255-4e2e-bc8a-b6c62c013c04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:31.533477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.533445 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-token" pod "239e461e-6255-4e2e-bc8a-b6c62c013c04" (UID: "239e461e-6255-4e2e-bc8a-b6c62c013c04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:31.533782 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.533752 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-podinfo" pod "239e461e-6255-4e2e-bc8a-b6c62c013c04" (UID: "239e461e-6255-4e2e-bc8a-b6c62c013c04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:25:31.534079 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.534058 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239e461e-6255-4e2e-bc8a-b6c62c013c04-kube-api-access-s6lp5" pod "239e461e-6255-4e2e-bc8a-b6c62c013c04" (UID: "239e461e-6255-4e2e-bc8a-b6c62c013c04"). InnerVolumeSpecName "kube-api-access-s6lp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:31.634125 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.633687 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:31.634125 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.633921 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:31.634125 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.633939 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239e461e-6255-4e2e-bc8a-b6c62c013c04-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:31.634125 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.634042 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:31.634125 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.634060 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6lp5\" (UniqueName: \"kubernetes.io/projected/239e461e-6255-4e2e-bc8a-b6c62c013c04-kube-api-access-s6lp5\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:31.634125 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.634075 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239e461e-6255-4e2e-bc8a-b6c62c013c04-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:31.634125 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.634090 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239e461e-6255-4e2e-bc8a-b6c62c013c04-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:31.700741 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.700700 7010 generic.go:356] "Generic (PLEG): container finished" podID="6ea67e42-fdce-4f95-9b23-a62f6fab20dd" containerID="05c696ae8391022118f109b802d233fa8411ee26b30ad36afe0c5890f1846448" exitCode=0 Oct 30 13:25:31.700941 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.700854 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" event={"ID":"6ea67e42-fdce-4f95-9b23-a62f6fab20dd","Type":"ContainerDied","Data":"05c696ae8391022118f109b802d233fa8411ee26b30ad36afe0c5890f1846448"} Oct 30 13:25:31.702962 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.702929 7010 generic.go:356] "Generic (PLEG): container finished" podID="239e461e-6255-4e2e-bc8a-b6c62c013c04" containerID="b01c853ba4b158563b65f2a24374db2916364468d17630d3f8a516e286196e2a" exitCode=0 Oct 30 13:25:31.703130 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.703011 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" event={"ID":"239e461e-6255-4e2e-bc8a-b6c62c013c04","Type":"ContainerDied","Data":"b01c853ba4b158563b65f2a24374db2916364468d17630d3f8a516e286196e2a"} Oct 30 13:25:31.703130 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.703068 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" event={"ID":"239e461e-6255-4e2e-bc8a-b6c62c013c04","Type":"ContainerDied","Data":"4211434e3c70ac173c7bfe13f92ed87c7ba53a6d0421495ad163548dc01eee6b"} Oct 30 13:25:31.703130 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.703099 7010 scope.go:122] "RemoveContainer" containerID="b01c853ba4b158563b65f2a24374db2916364468d17630d3f8a516e286196e2a" Oct 30 13:25:31.703996 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.703790 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg" Oct 30 13:25:31.716842 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.716802 7010 scope.go:122] "RemoveContainer" containerID="b01c853ba4b158563b65f2a24374db2916364468d17630d3f8a516e286196e2a" Oct 30 13:25:31.717243 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:25:31.717214 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b01c853ba4b158563b65f2a24374db2916364468d17630d3f8a516e286196e2a\": container with ID starting with b01c853ba4b158563b65f2a24374db2916364468d17630d3f8a516e286196e2a not found: ID does not exist" containerID="b01c853ba4b158563b65f2a24374db2916364468d17630d3f8a516e286196e2a" Oct 30 13:25:31.717343 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.717260 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01c853ba4b158563b65f2a24374db2916364468d17630d3f8a516e286196e2a"} err="failed to get container status \"b01c853ba4b158563b65f2a24374db2916364468d17630d3f8a516e286196e2a\": rpc error: code = NotFound desc = could not find container \"b01c853ba4b158563b65f2a24374db2916364468d17630d3f8a516e286196e2a\": container with ID starting with b01c853ba4b158563b65f2a24374db2916364468d17630d3f8a516e286196e2a not found: ID does not exist" Oct 30 13:25:31.755939 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.755911 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:25:31.761194 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.761161 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg"] Oct 30 13:25:31.764819 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.764792 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-zq6qg"] Oct 30 13:25:31.837262 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.837226 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-data\") pod \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " Oct 30 13:25:31.837478 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.837298 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-envoy\") pod \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " Oct 30 13:25:31.837478 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.837380 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-token\" (UniqueName: \"kubernetes.io/projected/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-token\") pod \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " Oct 30 13:25:31.837478 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.837472 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-workload-socket\") pod \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " Oct 30 13:25:31.837805 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.837532 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-kube-api-access-snwwf\" (UniqueName: \"kubernetes.io/projected/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-kube-api-access-snwwf\") pod \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " Oct 30 13:25:31.837805 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.837585 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-credential-socket\") pod \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " Oct 30 13:25:31.837805 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.837588 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-data" pod "6ea67e42-fdce-4f95-9b23-a62f6fab20dd" (UID: "6ea67e42-fdce-4f95-9b23-a62f6fab20dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:31.837805 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.837672 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istiod-ca-cert\") pod \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " Oct 30 13:25:31.837805 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.837713 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-podinfo\") pod \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " Oct 30 13:25:31.837805 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.837778 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-workload-certs\") pod \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\" (UID: \"6ea67e42-fdce-4f95-9b23-a62f6fab20dd\") " Oct 30 13:25:31.838127 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.838090 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-credential-socket" pod "6ea67e42-fdce-4f95-9b23-a62f6fab20dd" (UID: "6ea67e42-fdce-4f95-9b23-a62f6fab20dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:31.838127 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.838106 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-workload-certs" pod "6ea67e42-fdce-4f95-9b23-a62f6fab20dd" (UID: "6ea67e42-fdce-4f95-9b23-a62f6fab20dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:31.838241 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.838178 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-workload-socket" pod "6ea67e42-fdce-4f95-9b23-a62f6fab20dd" (UID: "6ea67e42-fdce-4f95-9b23-a62f6fab20dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:31.838342 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.838320 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istiod-ca-cert" pod "6ea67e42-fdce-4f95-9b23-a62f6fab20dd" (UID: "6ea67e42-fdce-4f95-9b23-a62f6fab20dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:31.838637 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.838614 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:31.838733 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.838642 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:31.838733 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.838659 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:31.838733 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.838673 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:31.838733 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.838685 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:31.840941 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.840910 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-envoy" pod "6ea67e42-fdce-4f95-9b23-a62f6fab20dd" (UID: "6ea67e42-fdce-4f95-9b23-a62f6fab20dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:31.841192 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.841164 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-kube-api-access-snwwf" pod "6ea67e42-fdce-4f95-9b23-a62f6fab20dd" (UID: "6ea67e42-fdce-4f95-9b23-a62f6fab20dd"). InnerVolumeSpecName "kube-api-access-snwwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:31.841466 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.841415 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-token" pod "6ea67e42-fdce-4f95-9b23-a62f6fab20dd" (UID: "6ea67e42-fdce-4f95-9b23-a62f6fab20dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:31.841466 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.841450 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-podinfo" pod "6ea67e42-fdce-4f95-9b23-a62f6fab20dd" (UID: "6ea67e42-fdce-4f95-9b23-a62f6fab20dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:25:31.940725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.940636 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snwwf\" (UniqueName: \"kubernetes.io/projected/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-kube-api-access-snwwf\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:31.940725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.940666 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:31.940725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.940676 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:31.940725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:31.940685 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea67e42-fdce-4f95-9b23-a62f6fab20dd-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:32.340351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.340306 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" podUID="7671398e-bfd7-4774-ac0e-eacef202eab2" containerName="payload-processing" containerID="cri-o://7425f392dd65af14e350be65d5f2aef2abbf7e07134e59de595c0312b17b7311" gracePeriod=25 Oct 30 13:25:32.501259 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.501233 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:25:32.650623 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.650469 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-token\" (UniqueName: \"kubernetes.io/projected/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-token\") pod \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " Oct 30 13:25:32.650842 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.650705 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-kube-api-access-h7fpk\" (UniqueName: \"kubernetes.io/projected/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-kube-api-access-h7fpk\") pod \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " Oct 30 13:25:32.650842 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.650822 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-credential-socket\") pod \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " Oct 30 13:25:32.650964 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.650880 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-data\") pod \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " Oct 30 13:25:32.651021 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.650963 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-workload-socket\") pod \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " Oct 30 13:25:32.651021 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.650998 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-envoy\") pod \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " Oct 30 13:25:32.651300 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.651163 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-credential-socket" pod "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" (UID: "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:32.651300 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.651200 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-podinfo\") pod \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " Oct 30 13:25:32.651300 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.651213 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-data" pod "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" (UID: "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:32.651300 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.651270 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-workload-socket" pod "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" (UID: "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:32.651721 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.651328 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-workload-certs\") pod \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " Oct 30 13:25:32.651721 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.651430 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istiod-ca-cert\") pod \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\" (UID: \"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d\") " Oct 30 13:25:32.651721 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.651550 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-workload-certs" pod "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" (UID: "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:32.651936 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.651904 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istiod-ca-cert" pod "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" (UID: "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:32.652540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.652509 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:32.652729 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.652676 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:32.652729 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.652704 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:32.652729 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.652719 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:32.652729 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.652733 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:32.653783 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.653288 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-token" pod "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" (UID: "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:32.653783 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.653647 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-envoy" pod "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" (UID: "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:25:32.653783 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.653763 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-podinfo" pod "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" (UID: "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:25:32.655151 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.655126 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-kube-api-access-h7fpk" pod "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" (UID: "ce7a5fa5-c1d8-4556-99f1-a0481dafc16d"). InnerVolumeSpecName "kube-api-access-h7fpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:32.715950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.715913 7010 generic.go:356] "Generic (PLEG): container finished" podID="ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" containerID="9e3ee119c33c324fa4a1a97fd09e6ed29cedaa2c02f76649e64162494bf2498c" exitCode=0 Oct 30 13:25:32.715950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.715948 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" event={"ID":"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d","Type":"ContainerDied","Data":"9e3ee119c33c324fa4a1a97fd09e6ed29cedaa2c02f76649e64162494bf2498c"} Oct 30 13:25:32.716162 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.715960 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" Oct 30 13:25:32.716162 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.716007 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6" event={"ID":"ce7a5fa5-c1d8-4556-99f1-a0481dafc16d","Type":"ContainerDied","Data":"d0bdb046011632c4237e406ad9dc0f430787ff122d2938e3132ef4ccd26da9a8"} Oct 30 13:25:32.716162 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.716036 7010 scope.go:122] "RemoveContainer" containerID="9e3ee119c33c324fa4a1a97fd09e6ed29cedaa2c02f76649e64162494bf2498c" Oct 30 13:25:32.717752 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.717726 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" event={"ID":"6ea67e42-fdce-4f95-9b23-a62f6fab20dd","Type":"ContainerDied","Data":"36881eb748c9f411b1dd37b8fc6e9df1c21a5de1e11ee8881aefe29a1b4ea0b0"} Oct 30 13:25:32.717876 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.717763 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4" Oct 30 13:25:32.728903 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.728882 7010 scope.go:122] "RemoveContainer" containerID="9e3ee119c33c324fa4a1a97fd09e6ed29cedaa2c02f76649e64162494bf2498c" Oct 30 13:25:32.729236 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:25:32.729212 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e3ee119c33c324fa4a1a97fd09e6ed29cedaa2c02f76649e64162494bf2498c\": container with ID starting with 9e3ee119c33c324fa4a1a97fd09e6ed29cedaa2c02f76649e64162494bf2498c not found: ID does not exist" containerID="9e3ee119c33c324fa4a1a97fd09e6ed29cedaa2c02f76649e64162494bf2498c" Oct 30 13:25:32.729367 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.729246 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3ee119c33c324fa4a1a97fd09e6ed29cedaa2c02f76649e64162494bf2498c"} err="failed to get container status \"9e3ee119c33c324fa4a1a97fd09e6ed29cedaa2c02f76649e64162494bf2498c\": rpc error: code = NotFound desc = could not find container \"9e3ee119c33c324fa4a1a97fd09e6ed29cedaa2c02f76649e64162494bf2498c\": container with ID starting with 9e3ee119c33c324fa4a1a97fd09e6ed29cedaa2c02f76649e64162494bf2498c not found: ID does not exist" Oct 30 13:25:32.729367 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.729265 7010 scope.go:122] "RemoveContainer" containerID="05c696ae8391022118f109b802d233fa8411ee26b30ad36afe0c5890f1846448" Oct 30 13:25:32.754008 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.753971 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7fpk\" (UniqueName: \"kubernetes.io/projected/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-kube-api-access-h7fpk\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:32.754008 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.754004 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:32.754008 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.754015 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:32.754250 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.754023 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:32.772528 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.772478 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6"] Oct 30 13:25:32.776776 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.776735 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-bvrm6"] Oct 30 13:25:32.811878 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.811841 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4"] Oct 30 13:25:32.819011 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:32.818956 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d7e62b-openshift-default-9d7d48765-5p6n4"] Oct 30 13:25:33.490408 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:33.490371 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="239e461e-6255-4e2e-bc8a-b6c62c013c04" path="/var/lib/kubelet/pods/239e461e-6255-4e2e-bc8a-b6c62c013c04/volumes" Oct 30 13:25:33.490961 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:33.490944 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ea67e42-fdce-4f95-9b23-a62f6fab20dd" path="/var/lib/kubelet/pods/6ea67e42-fdce-4f95-9b23-a62f6fab20dd/volumes" Oct 30 13:25:33.491400 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:33.491387 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" path="/var/lib/kubelet/pods/ce7a5fa5-c1d8-4556-99f1-a0481dafc16d/volumes" Oct 30 13:25:35.239277 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:35.239191 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" podUID="7671398e-bfd7-4774-ac0e-eacef202eab2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:25:45.239386 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:45.239247 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" podUID="7671398e-bfd7-4774-ac0e-eacef202eab2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:25:52.634256 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:52.634220 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" Oct 30 13:25:52.683461 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:52.683427 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7671398e-bfd7-4774-ac0e-eacef202eab2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7671398e-bfd7-4774-ac0e-eacef202eab2-plugins-config-volume\") pod \"7671398e-bfd7-4774-ac0e-eacef202eab2\" (UID: \"7671398e-bfd7-4774-ac0e-eacef202eab2\") " Oct 30 13:25:52.683670 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:52.683531 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7671398e-bfd7-4774-ac0e-eacef202eab2-kube-api-access-wxmrf\" (UniqueName: \"kubernetes.io/projected/7671398e-bfd7-4774-ac0e-eacef202eab2-kube-api-access-wxmrf\") pod \"7671398e-bfd7-4774-ac0e-eacef202eab2\" (UID: \"7671398e-bfd7-4774-ac0e-eacef202eab2\") " Oct 30 13:25:52.683867 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:52.683843 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7671398e-bfd7-4774-ac0e-eacef202eab2-plugins-config-volume" pod "7671398e-bfd7-4774-ac0e-eacef202eab2" (UID: "7671398e-bfd7-4774-ac0e-eacef202eab2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:25:52.684176 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:52.684155 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7671398e-bfd7-4774-ac0e-eacef202eab2-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:52.686087 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:52.686052 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7671398e-bfd7-4774-ac0e-eacef202eab2-kube-api-access-wxmrf" pod "7671398e-bfd7-4774-ac0e-eacef202eab2" (UID: "7671398e-bfd7-4774-ac0e-eacef202eab2"). InnerVolumeSpecName "kube-api-access-wxmrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:25:52.785339 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:52.785297 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxmrf\" (UniqueName: \"kubernetes.io/projected/7671398e-bfd7-4774-ac0e-eacef202eab2-kube-api-access-wxmrf\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:25:52.889535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:52.889481 7010 generic.go:356] "Generic (PLEG): container finished" podID="7671398e-bfd7-4774-ac0e-eacef202eab2" containerID="7425f392dd65af14e350be65d5f2aef2abbf7e07134e59de595c0312b17b7311" exitCode=0 Oct 30 13:25:52.889535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:52.889517 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" event={"ID":"7671398e-bfd7-4774-ac0e-eacef202eab2","Type":"ContainerDied","Data":"7425f392dd65af14e350be65d5f2aef2abbf7e07134e59de595c0312b17b7311"} Oct 30 13:25:52.889774 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:52.889546 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" Oct 30 13:25:52.889774 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:52.889579 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw" event={"ID":"7671398e-bfd7-4774-ac0e-eacef202eab2","Type":"ContainerDied","Data":"a9a79a2787534f61ef4bd53508ea0df68f367facbee5c77dc7c452300fbbe4c6"} Oct 30 13:25:52.889774 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:52.889606 7010 scope.go:122] "RemoveContainer" containerID="7425f392dd65af14e350be65d5f2aef2abbf7e07134e59de595c0312b17b7311" Oct 30 13:25:52.902652 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:52.902627 7010 scope.go:122] "RemoveContainer" containerID="7425f392dd65af14e350be65d5f2aef2abbf7e07134e59de595c0312b17b7311" Oct 30 13:25:52.902984 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:25:52.902957 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7425f392dd65af14e350be65d5f2aef2abbf7e07134e59de595c0312b17b7311\": container with ID starting with 7425f392dd65af14e350be65d5f2aef2abbf7e07134e59de595c0312b17b7311 not found: ID does not exist" containerID="7425f392dd65af14e350be65d5f2aef2abbf7e07134e59de595c0312b17b7311" Oct 30 13:25:52.903086 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:52.902994 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7425f392dd65af14e350be65d5f2aef2abbf7e07134e59de595c0312b17b7311"} err="failed to get container status \"7425f392dd65af14e350be65d5f2aef2abbf7e07134e59de595c0312b17b7311\": rpc error: code = NotFound desc = could not find container \"7425f392dd65af14e350be65d5f2aef2abbf7e07134e59de595c0312b17b7311\": container with ID starting with 7425f392dd65af14e350be65d5f2aef2abbf7e07134e59de595c0312b17b7311 not found: ID does not exist" Oct 30 13:25:52.945335 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:52.945302 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw"] Oct 30 13:25:52.952015 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:52.951984 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ddb271-579f67ccb6-92jmw"] Oct 30 13:25:53.490329 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:25:53.490296 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7671398e-bfd7-4774-ac0e-eacef202eab2" path="/var/lib/kubelet/pods/7671398e-bfd7-4774-ac0e-eacef202eab2/volumes" Oct 30 13:25:59.413188 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:25:59.413132 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:26:05.753402 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:05.753358 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d"] Oct 30 13:26:12.373753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:12.373714 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7"] Oct 30 13:26:12.374265 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:12.374011 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" podUID="e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" containerName="istio-proxy" containerID="cri-o://115ca13f1368588b7fe939059b58eae2947469d984b19c219ee1f7af523fec25" gracePeriod=30 Oct 30 13:26:16.584204 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:16.584166 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9"] Oct 30 13:26:17.670427 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.670398 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:26:17.783806 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.783758 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-token\" (UniqueName: \"kubernetes.io/projected/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-token\") pod \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " Oct 30 13:26:17.784011 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.783913 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-data\") pod \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " Oct 30 13:26:17.784091 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.784076 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-credential-socket\") pod \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " Oct 30 13:26:17.784189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.784161 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istiod-ca-cert\") pod \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " Oct 30 13:26:17.784260 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.784220 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-podinfo\") pod \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " Oct 30 13:26:17.784333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.784310 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-workload-certs\") pod \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " Oct 30 13:26:17.784392 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.784364 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-workload-socket\") pod \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " Oct 30 13:26:17.784451 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.784408 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-kube-api-access-88kq6\" (UniqueName: \"kubernetes.io/projected/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-kube-api-access-88kq6\") pod \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " Oct 30 13:26:17.784537 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.784444 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-data" pod "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" (UID: "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:17.784602 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.784530 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-credential-socket" pod "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" (UID: "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:17.784602 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.784546 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-envoy\") pod \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\" (UID: \"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d\") " Oct 30 13:26:17.784831 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.784806 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-workload-socket" pod "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" (UID: "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:17.785533 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.785463 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-workload-certs" pod "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" (UID: "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:17.785646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.785535 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:17.785646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.785581 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:17.785646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.785597 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:17.785793 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.785721 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istiod-ca-cert" pod "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" (UID: "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:17.788140 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.788114 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-token" pod "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" (UID: "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:17.788250 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.788152 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-podinfo" pod "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" (UID: "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:26:17.788250 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.788234 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-kube-api-access-88kq6" pod "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" (UID: "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d"). InnerVolumeSpecName "kube-api-access-88kq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:17.788771 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.788743 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-envoy" pod "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" (UID: "e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:17.887344 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.887303 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:17.887344 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.887339 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88kq6\" (UniqueName: \"kubernetes.io/projected/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-kube-api-access-88kq6\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:17.887344 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.887350 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:17.887344 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.887359 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:17.887679 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.887368 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:17.887679 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:17.887375 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:18.089302 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:18.089211 7010 generic.go:356] "Generic (PLEG): container finished" podID="e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" containerID="115ca13f1368588b7fe939059b58eae2947469d984b19c219ee1f7af523fec25" exitCode=0 Oct 30 13:26:18.089302 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:18.089264 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" Oct 30 13:26:18.089540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:18.089313 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" event={"ID":"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d","Type":"ContainerDied","Data":"115ca13f1368588b7fe939059b58eae2947469d984b19c219ee1f7af523fec25"} Oct 30 13:26:18.089540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:18.089375 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7" event={"ID":"e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d","Type":"ContainerDied","Data":"c66ded16f623bd72de2239914dbe733675f165121d7bb5f4b4dfb7093ecc8fd7"} Oct 30 13:26:18.089540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:18.089429 7010 scope.go:122] "RemoveContainer" containerID="115ca13f1368588b7fe939059b58eae2947469d984b19c219ee1f7af523fec25" Oct 30 13:26:18.103843 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:18.103818 7010 scope.go:122] "RemoveContainer" containerID="115ca13f1368588b7fe939059b58eae2947469d984b19c219ee1f7af523fec25" Oct 30 13:26:18.104173 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:26:18.104148 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115ca13f1368588b7fe939059b58eae2947469d984b19c219ee1f7af523fec25\": container with ID starting with 115ca13f1368588b7fe939059b58eae2947469d984b19c219ee1f7af523fec25 not found: ID does not exist" containerID="115ca13f1368588b7fe939059b58eae2947469d984b19c219ee1f7af523fec25" Oct 30 13:26:18.104231 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:18.104183 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115ca13f1368588b7fe939059b58eae2947469d984b19c219ee1f7af523fec25"} err="failed to get container status \"115ca13f1368588b7fe939059b58eae2947469d984b19c219ee1f7af523fec25\": rpc error: code = NotFound desc = could not find container \"115ca13f1368588b7fe939059b58eae2947469d984b19c219ee1f7af523fec25\": container with ID starting with 115ca13f1368588b7fe939059b58eae2947469d984b19c219ee1f7af523fec25 not found: ID does not exist" Oct 30 13:26:18.148621 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:18.148575 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7"] Oct 30 13:26:18.161152 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:18.161118 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-cxwc7"] Oct 30 13:26:18.413934 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:18.413887 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm"] Oct 30 13:26:18.414444 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:18.414388 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" podUID="a153c1e7-067c-426b-90df-87fa6649d078" containerName="maas-api" containerID="cri-o://5c41d604704cf084a0769f96cfacc9133fd618876aa5ca2f114c79c9ff1114b8" gracePeriod=30 Oct 30 13:26:19.101102 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:19.101063 7010 generic.go:356] "Generic (PLEG): container finished" podID="a153c1e7-067c-426b-90df-87fa6649d078" containerID="5c41d604704cf084a0769f96cfacc9133fd618876aa5ca2f114c79c9ff1114b8" exitCode=0 Oct 30 13:26:19.101633 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:19.101279 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" event={"ID":"a153c1e7-067c-426b-90df-87fa6649d078","Type":"ContainerDied","Data":"5c41d604704cf084a0769f96cfacc9133fd618876aa5ca2f114c79c9ff1114b8"} Oct 30 13:26:19.221346 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:19.221270 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:26:19.312108 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:19.312066 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a153c1e7-067c-426b-90df-87fa6649d078-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a153c1e7-067c-426b-90df-87fa6649d078-maas-api-tls\") pod \"a153c1e7-067c-426b-90df-87fa6649d078\" (UID: \"a153c1e7-067c-426b-90df-87fa6649d078\") " Oct 30 13:26:19.312367 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:19.312274 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a153c1e7-067c-426b-90df-87fa6649d078-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a153c1e7-067c-426b-90df-87fa6649d078-openshift-service-ca\") pod \"a153c1e7-067c-426b-90df-87fa6649d078\" (UID: \"a153c1e7-067c-426b-90df-87fa6649d078\") " Oct 30 13:26:19.312532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:19.312455 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a153c1e7-067c-426b-90df-87fa6649d078-kube-api-access-fhmc8\" (UniqueName: \"kubernetes.io/projected/a153c1e7-067c-426b-90df-87fa6649d078-kube-api-access-fhmc8\") pod \"a153c1e7-067c-426b-90df-87fa6649d078\" (UID: \"a153c1e7-067c-426b-90df-87fa6649d078\") " Oct 30 13:26:19.312532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:19.312513 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a153c1e7-067c-426b-90df-87fa6649d078-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a153c1e7-067c-426b-90df-87fa6649d078-metrics-certs\") pod \"a153c1e7-067c-426b-90df-87fa6649d078\" (UID: \"a153c1e7-067c-426b-90df-87fa6649d078\") " Oct 30 13:26:19.312909 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:19.312753 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a153c1e7-067c-426b-90df-87fa6649d078-openshift-service-ca" pod "a153c1e7-067c-426b-90df-87fa6649d078" (UID: "a153c1e7-067c-426b-90df-87fa6649d078"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:19.313148 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:19.313108 7010 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a153c1e7-067c-426b-90df-87fa6649d078-openshift-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:19.316353 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:19.315043 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a153c1e7-067c-426b-90df-87fa6649d078-maas-api-tls" pod "a153c1e7-067c-426b-90df-87fa6649d078" (UID: "a153c1e7-067c-426b-90df-87fa6649d078"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:19.316353 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:19.316156 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a153c1e7-067c-426b-90df-87fa6649d078-metrics-certs" pod "a153c1e7-067c-426b-90df-87fa6649d078" (UID: "a153c1e7-067c-426b-90df-87fa6649d078"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:19.323038 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:19.323003 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a153c1e7-067c-426b-90df-87fa6649d078-kube-api-access-fhmc8" pod "a153c1e7-067c-426b-90df-87fa6649d078" (UID: "a153c1e7-067c-426b-90df-87fa6649d078"). InnerVolumeSpecName "kube-api-access-fhmc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:19.414385 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:19.414352 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a153c1e7-067c-426b-90df-87fa6649d078-maas-api-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:19.414385 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:19.414382 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhmc8\" (UniqueName: \"kubernetes.io/projected/a153c1e7-067c-426b-90df-87fa6649d078-kube-api-access-fhmc8\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:19.414385 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:19.414391 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a153c1e7-067c-426b-90df-87fa6649d078-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:19.491341 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:19.491253 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" path="/var/lib/kubelet/pods/e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d/volumes" Oct 30 13:26:20.110217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:20.110181 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" Oct 30 13:26:20.110757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:20.110221 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm" event={"ID":"a153c1e7-067c-426b-90df-87fa6649d078","Type":"ContainerDied","Data":"0e2492f8f0c948910e9e548bdc51e8ff536039d11dffaf2b2c2f0d1e6a93816e"} Oct 30 13:26:20.110757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:20.110292 7010 scope.go:122] "RemoveContainer" containerID="5c41d604704cf084a0769f96cfacc9133fd618876aa5ca2f114c79c9ff1114b8" Oct 30 13:26:20.188626 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:20.188587 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm"] Oct 30 13:26:20.201750 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:20.201710 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31721c-5b5f798fb8-g5txm"] Oct 30 13:26:20.775610 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:20.775573 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" podUID="6b9c833b-eab4-42d4-bda7-fac3c9c7455c" containerName="main" containerID="cri-o://9fdc414eb2516b9772fc0e96b8b057c02f20ad8f19c323f52c649c789d5c3e1a" gracePeriod=45 Oct 30 13:26:21.081385 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.081357 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:26:21.120077 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.120039 7010 generic.go:356] "Generic (PLEG): container finished" podID="6b9c833b-eab4-42d4-bda7-fac3c9c7455c" containerID="9fdc414eb2516b9772fc0e96b8b057c02f20ad8f19c323f52c649c789d5c3e1a" exitCode=0 Oct 30 13:26:21.120077 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.120079 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" Oct 30 13:26:21.120605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.120118 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" event={"ID":"6b9c833b-eab4-42d4-bda7-fac3c9c7455c","Type":"ContainerDied","Data":"9fdc414eb2516b9772fc0e96b8b057c02f20ad8f19c323f52c649c789d5c3e1a"} Oct 30 13:26:21.120605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.120162 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d" event={"ID":"6b9c833b-eab4-42d4-bda7-fac3c9c7455c","Type":"ContainerDied","Data":"1fe5b9ef1c457aa547788cdffa5b8cc75dfe0d4b9b1a32daf406516912f3fec0"} Oct 30 13:26:21.120605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.120183 7010 scope.go:122] "RemoveContainer" containerID="9fdc414eb2516b9772fc0e96b8b057c02f20ad8f19c323f52c649c789d5c3e1a" Oct 30 13:26:21.132081 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.132058 7010 scope.go:122] "RemoveContainer" containerID="9fdc414eb2516b9772fc0e96b8b057c02f20ad8f19c323f52c649c789d5c3e1a" Oct 30 13:26:21.132472 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:26:21.132440 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fdc414eb2516b9772fc0e96b8b057c02f20ad8f19c323f52c649c789d5c3e1a\": container with ID starting with 9fdc414eb2516b9772fc0e96b8b057c02f20ad8f19c323f52c649c789d5c3e1a not found: ID does not exist" containerID="9fdc414eb2516b9772fc0e96b8b057c02f20ad8f19c323f52c649c789d5c3e1a" Oct 30 13:26:21.132605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.132483 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fdc414eb2516b9772fc0e96b8b057c02f20ad8f19c323f52c649c789d5c3e1a"} err="failed to get container status \"9fdc414eb2516b9772fc0e96b8b057c02f20ad8f19c323f52c649c789d5c3e1a\": rpc error: code = NotFound desc = could not find container \"9fdc414eb2516b9772fc0e96b8b057c02f20ad8f19c323f52c649c789d5c3e1a\": container with ID starting with 9fdc414eb2516b9772fc0e96b8b057c02f20ad8f19c323f52c649c789d5c3e1a not found: ID does not exist" Oct 30 13:26:21.138019 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.137989 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-dshm\") pod \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " Oct 30 13:26:21.138202 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.138033 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-kube-api-access-78sd7\" (UniqueName: \"kubernetes.io/projected/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-kube-api-access-78sd7\") pod \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " Oct 30 13:26:21.138202 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.138191 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-model-cache\") pod \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " Oct 30 13:26:21.138330 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.138222 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-tmp-dir\") pod \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " Oct 30 13:26:21.138330 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.138299 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-tls-certs\") pod \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " Oct 30 13:26:21.138443 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.138356 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-home\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-home\") pod \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\" (UID: \"6b9c833b-eab4-42d4-bda7-fac3c9c7455c\") " Oct 30 13:26:21.138866 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.138634 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-tmp-dir" pod "6b9c833b-eab4-42d4-bda7-fac3c9c7455c" (UID: "6b9c833b-eab4-42d4-bda7-fac3c9c7455c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:21.138866 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.138808 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-model-cache" pod "6b9c833b-eab4-42d4-bda7-fac3c9c7455c" (UID: "6b9c833b-eab4-42d4-bda7-fac3c9c7455c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:21.138866 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.138834 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-home" pod "6b9c833b-eab4-42d4-bda7-fac3c9c7455c" (UID: "6b9c833b-eab4-42d4-bda7-fac3c9c7455c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:21.139177 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.139156 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:21.139252 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.139182 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:21.139252 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.139198 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:21.143074 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.141881 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-dshm" pod "6b9c833b-eab4-42d4-bda7-fac3c9c7455c" (UID: "6b9c833b-eab4-42d4-bda7-fac3c9c7455c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:21.143074 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.141948 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-kube-api-access-78sd7" pod "6b9c833b-eab4-42d4-bda7-fac3c9c7455c" (UID: "6b9c833b-eab4-42d4-bda7-fac3c9c7455c"). InnerVolumeSpecName "kube-api-access-78sd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:21.143424 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.143397 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-tls-certs" pod "6b9c833b-eab4-42d4-bda7-fac3c9c7455c" (UID: "6b9c833b-eab4-42d4-bda7-fac3c9c7455c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:21.240664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.240632 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:21.240664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.240661 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78sd7\" (UniqueName: \"kubernetes.io/projected/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-kube-api-access-78sd7\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:21.240865 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.240673 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b9c833b-eab4-42d4-bda7-fac3c9c7455c-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:21.513504 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.513447 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a153c1e7-067c-426b-90df-87fa6649d078" path="/var/lib/kubelet/pods/a153c1e7-067c-426b-90df-87fa6649d078/volumes" Oct 30 13:26:21.524372 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.524127 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d"] Oct 30 13:26:21.524372 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.524180 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5dd8cc/test-model-5dd8cc-kserve-64fb5df7df-mkw6d"] Oct 30 13:26:21.585016 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:21.584944 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" podUID="fdb4594d-71b9-405c-b1eb-654a082160e8" containerName="payload-processing" containerID="cri-o://d26f2d777f13135af61b7b065022bdaca17b578d66ef60fb5508e537d1d2c5e7" gracePeriod=25 Oct 30 13:26:22.694369 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:22.693898 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt"] Oct 30 13:26:23.491341 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:23.491307 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b9c833b-eab4-42d4-bda7-fac3c9c7455c" path="/var/lib/kubelet/pods/6b9c833b-eab4-42d4-bda7-fac3c9c7455c/volumes" Oct 30 13:26:29.271228 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:29.271188 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" podUID="fdb4594d-71b9-405c-b1eb-654a082160e8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:26:29.446525 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:26:29.446079 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:26:34.271847 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:34.271803 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z"] Oct 30 13:26:37.590434 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:37.590401 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2"] Oct 30 13:26:37.735547 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:37.732729 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" podUID="81c361d4-8afa-4d5a-a827-ef4bf03a73c0" containerName="main" containerID="cri-o://15c2cfaeae3e26932c5cce7f3674c5d5df05f56d131e73f447a11950374e165c" gracePeriod=45 Oct 30 13:26:37.859871 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:37.859781 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" podUID="81c361d4-8afa-4d5a-a827-ef4bf03a73c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": dial tcp 10.134.0.107:8000: connect: connection refused" Oct 30 13:26:38.060670 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.060641 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:26:38.139674 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.139637 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-model-cache\") pod \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " Oct 30 13:26:38.139879 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.139714 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-kube-api-access-97c7r\" (UniqueName: \"kubernetes.io/projected/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-kube-api-access-97c7r\") pod \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " Oct 30 13:26:38.139879 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.139737 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-home\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-home\") pod \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " Oct 30 13:26:38.139879 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.139845 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-tmp-dir\") pod \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " Oct 30 13:26:38.140079 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.139881 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-tls-certs\") pod \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " Oct 30 13:26:38.140079 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.139907 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-dshm\") pod \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\" (UID: \"81c361d4-8afa-4d5a-a827-ef4bf03a73c0\") " Oct 30 13:26:38.140356 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.140269 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-model-cache" pod "81c361d4-8afa-4d5a-a827-ef4bf03a73c0" (UID: "81c361d4-8afa-4d5a-a827-ef4bf03a73c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:38.140931 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.140783 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:38.141222 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.141096 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-home" pod "81c361d4-8afa-4d5a-a827-ef4bf03a73c0" (UID: "81c361d4-8afa-4d5a-a827-ef4bf03a73c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:38.141382 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.141356 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-tmp-dir" pod "81c361d4-8afa-4d5a-a827-ef4bf03a73c0" (UID: "81c361d4-8afa-4d5a-a827-ef4bf03a73c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:38.143094 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.143064 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-kube-api-access-97c7r" pod "81c361d4-8afa-4d5a-a827-ef4bf03a73c0" (UID: "81c361d4-8afa-4d5a-a827-ef4bf03a73c0"). InnerVolumeSpecName "kube-api-access-97c7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:38.143787 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.143678 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-dshm" pod "81c361d4-8afa-4d5a-a827-ef4bf03a73c0" (UID: "81c361d4-8afa-4d5a-a827-ef4bf03a73c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:38.144608 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.144581 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-tls-certs" pod "81c361d4-8afa-4d5a-a827-ef4bf03a73c0" (UID: "81c361d4-8afa-4d5a-a827-ef4bf03a73c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:38.275509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.268206 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97c7r\" (UniqueName: \"kubernetes.io/projected/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-kube-api-access-97c7r\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:38.275509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.268242 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:38.275509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.268287 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:38.275509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.268302 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:38.275509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.268315 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81c361d4-8afa-4d5a-a827-ef4bf03a73c0-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:38.280833 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.280799 7010 generic.go:356] "Generic (PLEG): container finished" podID="81c361d4-8afa-4d5a-a827-ef4bf03a73c0" containerID="15c2cfaeae3e26932c5cce7f3674c5d5df05f56d131e73f447a11950374e165c" exitCode=0 Oct 30 13:26:38.280991 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.280849 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" Oct 30 13:26:38.280991 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.280878 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" event={"ID":"81c361d4-8afa-4d5a-a827-ef4bf03a73c0","Type":"ContainerDied","Data":"15c2cfaeae3e26932c5cce7f3674c5d5df05f56d131e73f447a11950374e165c"} Oct 30 13:26:38.280991 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.280922 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt" event={"ID":"81c361d4-8afa-4d5a-a827-ef4bf03a73c0","Type":"ContainerDied","Data":"2781682f4ae475a2204cb6d6071cf59ee1c24e46115d5ccd01d83d0d0db2e164"} Oct 30 13:26:38.280991 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.280946 7010 scope.go:122] "RemoveContainer" containerID="15c2cfaeae3e26932c5cce7f3674c5d5df05f56d131e73f447a11950374e165c" Oct 30 13:26:38.291965 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.291942 7010 scope.go:122] "RemoveContainer" containerID="15c2cfaeae3e26932c5cce7f3674c5d5df05f56d131e73f447a11950374e165c" Oct 30 13:26:38.292284 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:26:38.292266 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c2cfaeae3e26932c5cce7f3674c5d5df05f56d131e73f447a11950374e165c\": container with ID starting with 15c2cfaeae3e26932c5cce7f3674c5d5df05f56d131e73f447a11950374e165c not found: ID does not exist" containerID="15c2cfaeae3e26932c5cce7f3674c5d5df05f56d131e73f447a11950374e165c" Oct 30 13:26:38.292347 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.292299 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c2cfaeae3e26932c5cce7f3674c5d5df05f56d131e73f447a11950374e165c"} err="failed to get container status \"15c2cfaeae3e26932c5cce7f3674c5d5df05f56d131e73f447a11950374e165c\": rpc error: code = NotFound desc = could not find container \"15c2cfaeae3e26932c5cce7f3674c5d5df05f56d131e73f447a11950374e165c\": container with ID starting with 15c2cfaeae3e26932c5cce7f3674c5d5df05f56d131e73f447a11950374e165c not found: ID does not exist" Oct 30 13:26:38.340915 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.340881 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt"] Oct 30 13:26:38.343868 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:38.343839 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8467ed/test-model-8467ed-kserve-74bf9d887-stmrt"] Oct 30 13:26:39.270701 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:39.270655 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" podUID="fdb4594d-71b9-405c-b1eb-654a082160e8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:26:39.272671 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:39.272633 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" podUID="f572a662-3ef2-4230-a1f5-6bf0ae106344" containerName="payload-pre-processing" containerID="cri-o://ac8582d7c6917ae1fcebc6059e0a100ceca867cdb88a9d6d32563ef0f6139bd0" gracePeriod=25 Oct 30 13:26:39.493710 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:39.493667 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81c361d4-8afa-4d5a-a827-ef4bf03a73c0" path="/var/lib/kubelet/pods/81c361d4-8afa-4d5a-a827-ef4bf03a73c0/volumes" Oct 30 13:26:39.838175 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:39.838134 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" podUID="f572a662-3ef2-4230-a1f5-6bf0ae106344" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:26:41.887503 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:41.887463 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" Oct 30 13:26:41.923505 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:41.923454 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb4594d-71b9-405c-b1eb-654a082160e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb4594d-71b9-405c-b1eb-654a082160e8-plugins-config-volume\") pod \"fdb4594d-71b9-405c-b1eb-654a082160e8\" (UID: \"fdb4594d-71b9-405c-b1eb-654a082160e8\") " Oct 30 13:26:41.923702 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:41.923560 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb4594d-71b9-405c-b1eb-654a082160e8-kube-api-access-4z6b7\" (UniqueName: \"kubernetes.io/projected/fdb4594d-71b9-405c-b1eb-654a082160e8-kube-api-access-4z6b7\") pod \"fdb4594d-71b9-405c-b1eb-654a082160e8\" (UID: \"fdb4594d-71b9-405c-b1eb-654a082160e8\") " Oct 30 13:26:41.924055 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:41.924026 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb4594d-71b9-405c-b1eb-654a082160e8-plugins-config-volume" pod "fdb4594d-71b9-405c-b1eb-654a082160e8" (UID: "fdb4594d-71b9-405c-b1eb-654a082160e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:41.926196 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:41.926160 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb4594d-71b9-405c-b1eb-654a082160e8-kube-api-access-4z6b7" pod "fdb4594d-71b9-405c-b1eb-654a082160e8" (UID: "fdb4594d-71b9-405c-b1eb-654a082160e8"). InnerVolumeSpecName "kube-api-access-4z6b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:42.026262 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:42.026188 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb4594d-71b9-405c-b1eb-654a082160e8-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:42.026262 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:42.026225 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z6b7\" (UniqueName: \"kubernetes.io/projected/fdb4594d-71b9-405c-b1eb-654a082160e8-kube-api-access-4z6b7\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:42.316240 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:42.316198 7010 generic.go:356] "Generic (PLEG): container finished" podID="fdb4594d-71b9-405c-b1eb-654a082160e8" containerID="d26f2d777f13135af61b7b065022bdaca17b578d66ef60fb5508e537d1d2c5e7" exitCode=0 Oct 30 13:26:42.316477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:42.316251 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" Oct 30 13:26:42.316477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:42.316286 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" event={"ID":"fdb4594d-71b9-405c-b1eb-654a082160e8","Type":"ContainerDied","Data":"d26f2d777f13135af61b7b065022bdaca17b578d66ef60fb5508e537d1d2c5e7"} Oct 30 13:26:42.316477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:42.316391 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9" event={"ID":"fdb4594d-71b9-405c-b1eb-654a082160e8","Type":"ContainerDied","Data":"625e98846aef128e04014e19ac381d8cdac621e3f2bf22b2580081e188a3dd3b"} Oct 30 13:26:42.316477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:42.316424 7010 scope.go:122] "RemoveContainer" containerID="d26f2d777f13135af61b7b065022bdaca17b578d66ef60fb5508e537d1d2c5e7" Oct 30 13:26:42.327673 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:42.327649 7010 scope.go:122] "RemoveContainer" containerID="d26f2d777f13135af61b7b065022bdaca17b578d66ef60fb5508e537d1d2c5e7" Oct 30 13:26:42.327982 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:26:42.327960 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26f2d777f13135af61b7b065022bdaca17b578d66ef60fb5508e537d1d2c5e7\": container with ID starting with d26f2d777f13135af61b7b065022bdaca17b578d66ef60fb5508e537d1d2c5e7 not found: ID does not exist" containerID="d26f2d777f13135af61b7b065022bdaca17b578d66ef60fb5508e537d1d2c5e7" Oct 30 13:26:42.328045 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:42.327995 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26f2d777f13135af61b7b065022bdaca17b578d66ef60fb5508e537d1d2c5e7"} err="failed to get container status \"d26f2d777f13135af61b7b065022bdaca17b578d66ef60fb5508e537d1d2c5e7\": rpc error: code = NotFound desc = could not find container \"d26f2d777f13135af61b7b065022bdaca17b578d66ef60fb5508e537d1d2c5e7\": container with ID starting with d26f2d777f13135af61b7b065022bdaca17b578d66ef60fb5508e537d1d2c5e7 not found: ID does not exist" Oct 30 13:26:42.371103 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:42.371065 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9"] Oct 30 13:26:42.383900 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:42.383865 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-31721c-dbf7588f8-h42j9"] Oct 30 13:26:43.491831 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:43.491795 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb4594d-71b9-405c-b1eb-654a082160e8" path="/var/lib/kubelet/pods/fdb4594d-71b9-405c-b1eb-654a082160e8/volumes" Oct 30 13:26:49.839192 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:49.839142 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" podUID="f572a662-3ef2-4230-a1f5-6bf0ae106344" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:26:50.942187 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942130 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66"] Oct 30 13:26:50.942838 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942821 7010 cpu_manager.go:414] "containerMap: removing container" podUID="ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" containerName="istio-proxy" Oct 30 13:26:50.942890 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942840 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" containerName="istio-proxy" Oct 30 13:26:50.942890 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942858 7010 cpu_manager.go:414] "containerMap: removing container" podUID="239e461e-6255-4e2e-bc8a-b6c62c013c04" containerName="istio-proxy" Oct 30 13:26:50.942890 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942863 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="239e461e-6255-4e2e-bc8a-b6c62c013c04" containerName="istio-proxy" Oct 30 13:26:50.942890 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942879 7010 cpu_manager.go:414] "containerMap: removing container" podUID="7671398e-bfd7-4774-ac0e-eacef202eab2" containerName="payload-processing" Oct 30 13:26:50.942890 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942885 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7671398e-bfd7-4774-ac0e-eacef202eab2" containerName="payload-processing" Oct 30 13:26:50.943065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942894 7010 cpu_manager.go:414] "containerMap: removing container" podUID="81c361d4-8afa-4d5a-a827-ef4bf03a73c0" containerName="main" Oct 30 13:26:50.943065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942898 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81c361d4-8afa-4d5a-a827-ef4bf03a73c0" containerName="main" Oct 30 13:26:50.943065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942927 7010 cpu_manager.go:414] "containerMap: removing container" podUID="6ea67e42-fdce-4f95-9b23-a62f6fab20dd" containerName="istio-proxy" Oct 30 13:26:50.943065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942934 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea67e42-fdce-4f95-9b23-a62f6fab20dd" containerName="istio-proxy" Oct 30 13:26:50.943065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942948 7010 cpu_manager.go:414] "containerMap: removing container" podUID="96f80372-7a4d-464d-a85b-ad4fa0742324" containerName="main" Oct 30 13:26:50.943065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942952 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96f80372-7a4d-464d-a85b-ad4fa0742324" containerName="main" Oct 30 13:26:50.943065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942968 7010 cpu_manager.go:414] "containerMap: removing container" podUID="a153c1e7-067c-426b-90df-87fa6649d078" containerName="maas-api" Oct 30 13:26:50.943065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942973 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a153c1e7-067c-426b-90df-87fa6649d078" containerName="maas-api" Oct 30 13:26:50.943065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942982 7010 cpu_manager.go:414] "containerMap: removing container" podUID="fdb4594d-71b9-405c-b1eb-654a082160e8" containerName="payload-processing" Oct 30 13:26:50.943065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942986 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb4594d-71b9-405c-b1eb-654a082160e8" containerName="payload-processing" Oct 30 13:26:50.943065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.942996 7010 cpu_manager.go:414] "containerMap: removing container" podUID="e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" containerName="istio-proxy" Oct 30 13:26:50.943065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.943004 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" containerName="istio-proxy" Oct 30 13:26:50.943065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.943018 7010 cpu_manager.go:414] "containerMap: removing container" podUID="6b9c833b-eab4-42d4-bda7-fac3c9c7455c" containerName="main" Oct 30 13:26:50.943065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.943023 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9c833b-eab4-42d4-bda7-fac3c9c7455c" containerName="main" Oct 30 13:26:50.943681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.943162 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="81c361d4-8afa-4d5a-a827-ef4bf03a73c0" containerName="main" Oct 30 13:26:50.943681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.943187 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea67e42-fdce-4f95-9b23-a62f6fab20dd" containerName="istio-proxy" Oct 30 13:26:50.943681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.943199 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="96f80372-7a4d-464d-a85b-ad4fa0742324" containerName="main" Oct 30 13:26:50.943681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.943210 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="a153c1e7-067c-426b-90df-87fa6649d078" containerName="maas-api" Oct 30 13:26:50.943681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.943219 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb4594d-71b9-405c-b1eb-654a082160e8" containerName="payload-processing" Oct 30 13:26:50.943681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.943227 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3da8c6a-cc2f-4c0d-9b5a-d04027578b4d" containerName="istio-proxy" Oct 30 13:26:50.943681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.943234 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9c833b-eab4-42d4-bda7-fac3c9c7455c" containerName="main" Oct 30 13:26:50.943681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.943241 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce7a5fa5-c1d8-4556-99f1-a0481dafc16d" containerName="istio-proxy" Oct 30 13:26:50.943681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.943248 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="239e461e-6255-4e2e-bc8a-b6c62c013c04" containerName="istio-proxy" Oct 30 13:26:50.943681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.943255 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="7671398e-bfd7-4774-ac0e-eacef202eab2" containerName="payload-processing" Oct 30 13:26:50.947157 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.947132 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:50.966027 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.965990 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-47628d-openshift-default-dockercfg-pzxrk\"" Oct 30 13:26:50.983925 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:50.983885 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66"] Oct 30 13:26:51.067745 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.067707 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a68869-2b11-4796-a767-ab7402efbf87-istiod-ca-cert\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.067745 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.067748 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-credential-socket\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.067969 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.067928 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hxvk\" (UniqueName: \"kubernetes.io/projected/03a68869-2b11-4796-a767-ab7402efbf87-kube-api-access-6hxvk\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.067969 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.067952 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-istio-data\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.068038 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.067978 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a68869-2b11-4796-a767-ab7402efbf87-istio-podinfo\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.068038 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.068000 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-workload-certs\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.068143 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.068126 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a68869-2b11-4796-a767-ab7402efbf87-istio-token\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.068192 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.068179 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-workload-socket\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.068315 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.068291 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-istio-envoy\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.169897 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.169854 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hxvk\" (UniqueName: \"kubernetes.io/projected/03a68869-2b11-4796-a767-ab7402efbf87-kube-api-access-6hxvk\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.169897 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.169899 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-istio-data\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.170135 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.169925 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a68869-2b11-4796-a767-ab7402efbf87-istio-podinfo\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.170135 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.170048 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-workload-certs\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.170135 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.170108 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a68869-2b11-4796-a767-ab7402efbf87-istio-token\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.170273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.170152 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-workload-socket\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.170273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.170249 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-istio-envoy\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.170381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.170294 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-istio-data\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.170381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.170319 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a68869-2b11-4796-a767-ab7402efbf87-istiod-ca-cert\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.170381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.170350 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-credential-socket\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.170576 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.170386 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-workload-certs\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.170631 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.170598 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-workload-socket\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.170757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.170727 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-credential-socket\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.170977 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.170958 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a68869-2b11-4796-a767-ab7402efbf87-istiod-ca-cert\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.172536 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.172514 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a68869-2b11-4796-a767-ab7402efbf87-istio-podinfo\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.172631 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.172567 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-istio-envoy\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.194421 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.194332 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a68869-2b11-4796-a767-ab7402efbf87-istio-token\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.194580 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.194479 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hxvk\" (UniqueName: \"kubernetes.io/projected/03a68869-2b11-4796-a767-ab7402efbf87-kube-api-access-6hxvk\") pod \"e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.261800 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.261766 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:51.593642 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.593613 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66"] Oct 30 13:26:51.595320 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:26:51.595295 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a68869_2b11_4796_a767_ab7402efbf87.slice/crio-31d9108b250eaac4282d3102d881557bafe4a7258c5ee9cbcff1536c56e600e1 WatchSource:0}: Error finding container 31d9108b250eaac4282d3102d881557bafe4a7258c5ee9cbcff1536c56e600e1: Status 404 returned error can't find the container with id 31d9108b250eaac4282d3102d881557bafe4a7258c5ee9cbcff1536c56e600e1 Oct 30 13:26:51.598265 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.598232 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:26:51.598358 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.598306 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:26:51.598358 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:51.598336 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:26:52.423583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:52.423543 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" event={"ID":"03a68869-2b11-4796-a767-ab7402efbf87","Type":"ContainerStarted","Data":"46cc19fec274a9ef0339e3a5bbeb807b27f8599db8d87e9b0967756b0a53c20d"} Oct 30 13:26:52.424023 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:52.423638 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" event={"ID":"03a68869-2b11-4796-a767-ab7402efbf87","Type":"ContainerStarted","Data":"31d9108b250eaac4282d3102d881557bafe4a7258c5ee9cbcff1536c56e600e1"} Oct 30 13:26:52.513724 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:52.513643 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" podStartSLOduration=2.513620542 podStartE2EDuration="2.513620542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:52.482829803 +0000 UTC m=+2069.115016965" watchObservedRunningTime="2026-10-30 13:26:52.513620542 +0000 UTC m=+2069.145807733" Oct 30 13:26:52.613449 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:52.613406 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" podUID="fe53a3fb-1f3d-463b-a165-713c55449df7" containerName="main" containerID="cri-o://20ba82d1fb789aaa2ab3103fa870f97aad1b1a74473cdb882726111cdcd23bc6" gracePeriod=45 Oct 30 13:26:52.952257 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:52.952224 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:26:53.129104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.129075 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe53a3fb-1f3d-463b-a165-713c55449df7-kube-api-access-dmpmb\" (UniqueName: \"kubernetes.io/projected/fe53a3fb-1f3d-463b-a165-713c55449df7-kube-api-access-dmpmb\") pod \"fe53a3fb-1f3d-463b-a165-713c55449df7\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " Oct 30 13:26:53.129293 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.129157 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-model-cache\") pod \"fe53a3fb-1f3d-463b-a165-713c55449df7\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " Oct 30 13:26:53.129374 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.129355 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-home\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-home\") pod \"fe53a3fb-1f3d-463b-a165-713c55449df7\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " Oct 30 13:26:53.129469 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.129436 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-model-cache" pod "fe53a3fb-1f3d-463b-a165-713c55449df7" (UID: "fe53a3fb-1f3d-463b-a165-713c55449df7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:53.129588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.129574 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe53a3fb-1f3d-463b-a165-713c55449df7-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe53a3fb-1f3d-463b-a165-713c55449df7-tls-certs\") pod \"fe53a3fb-1f3d-463b-a165-713c55449df7\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " Oct 30 13:26:53.129677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.129604 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-tmp-dir\") pod \"fe53a3fb-1f3d-463b-a165-713c55449df7\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " Oct 30 13:26:53.129677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.129640 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-dshm\") pod \"fe53a3fb-1f3d-463b-a165-713c55449df7\" (UID: \"fe53a3fb-1f3d-463b-a165-713c55449df7\") " Oct 30 13:26:53.129846 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.129749 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-home" pod "fe53a3fb-1f3d-463b-a165-713c55449df7" (UID: "fe53a3fb-1f3d-463b-a165-713c55449df7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:53.129913 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.129871 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-tmp-dir" pod "fe53a3fb-1f3d-463b-a165-713c55449df7" (UID: "fe53a3fb-1f3d-463b-a165-713c55449df7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:53.130253 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.130232 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:53.130360 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.130254 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:53.130360 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.130269 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:53.132101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.132074 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe53a3fb-1f3d-463b-a165-713c55449df7-kube-api-access-dmpmb" pod "fe53a3fb-1f3d-463b-a165-713c55449df7" (UID: "fe53a3fb-1f3d-463b-a165-713c55449df7"). InnerVolumeSpecName "kube-api-access-dmpmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:53.132101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.132094 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe53a3fb-1f3d-463b-a165-713c55449df7-tls-certs" pod "fe53a3fb-1f3d-463b-a165-713c55449df7" (UID: "fe53a3fb-1f3d-463b-a165-713c55449df7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:26:53.132833 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.132806 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-dshm" pod "fe53a3fb-1f3d-463b-a165-713c55449df7" (UID: "fe53a3fb-1f3d-463b-a165-713c55449df7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:26:53.232159 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.232116 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe53a3fb-1f3d-463b-a165-713c55449df7-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:53.232159 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.232151 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe53a3fb-1f3d-463b-a165-713c55449df7-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:53.232159 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.232165 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmpmb\" (UniqueName: \"kubernetes.io/projected/fe53a3fb-1f3d-463b-a165-713c55449df7-kube-api-access-dmpmb\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:53.262321 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.262186 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:53.263955 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.263923 7010 patch_prober.go:29] interesting pod/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:26:53.264118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.264000 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" podUID="03a68869-2b11-4796-a767-ab7402efbf87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 13:26:53.431742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.431640 7010 generic.go:356] "Generic (PLEG): container finished" podID="fe53a3fb-1f3d-463b-a165-713c55449df7" containerID="20ba82d1fb789aaa2ab3103fa870f97aad1b1a74473cdb882726111cdcd23bc6" exitCode=0 Oct 30 13:26:53.431742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.431698 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" Oct 30 13:26:53.431742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.431724 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" event={"ID":"fe53a3fb-1f3d-463b-a165-713c55449df7","Type":"ContainerDied","Data":"20ba82d1fb789aaa2ab3103fa870f97aad1b1a74473cdb882726111cdcd23bc6"} Oct 30 13:26:53.432334 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.431770 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2" event={"ID":"fe53a3fb-1f3d-463b-a165-713c55449df7","Type":"ContainerDied","Data":"693807bd4c24d180dc95c07560f18321d725bbca4bd687e70f101f33fdb4e0b4"} Oct 30 13:26:53.432334 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.431789 7010 scope.go:122] "RemoveContainer" containerID="20ba82d1fb789aaa2ab3103fa870f97aad1b1a74473cdb882726111cdcd23bc6" Oct 30 13:26:53.442946 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.442925 7010 scope.go:122] "RemoveContainer" containerID="20ba82d1fb789aaa2ab3103fa870f97aad1b1a74473cdb882726111cdcd23bc6" Oct 30 13:26:53.443286 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:26:53.443265 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20ba82d1fb789aaa2ab3103fa870f97aad1b1a74473cdb882726111cdcd23bc6\": container with ID starting with 20ba82d1fb789aaa2ab3103fa870f97aad1b1a74473cdb882726111cdcd23bc6 not found: ID does not exist" containerID="20ba82d1fb789aaa2ab3103fa870f97aad1b1a74473cdb882726111cdcd23bc6" Oct 30 13:26:53.443338 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.443295 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20ba82d1fb789aaa2ab3103fa870f97aad1b1a74473cdb882726111cdcd23bc6"} err="failed to get container status \"20ba82d1fb789aaa2ab3103fa870f97aad1b1a74473cdb882726111cdcd23bc6\": rpc error: code = NotFound desc = could not find container \"20ba82d1fb789aaa2ab3103fa870f97aad1b1a74473cdb882726111cdcd23bc6\": container with ID starting with 20ba82d1fb789aaa2ab3103fa870f97aad1b1a74473cdb882726111cdcd23bc6 not found: ID does not exist" Oct 30 13:26:53.518625 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.518594 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2"] Oct 30 13:26:53.518803 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:53.518635 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ff4285/auth-test-model-ff4285-kserve-6fdf8546bf-p5rd2"] Oct 30 13:26:54.263002 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:54.262968 7010 patch_prober.go:29] interesting pod/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:26:54.263197 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:54.263023 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" podUID="03a68869-2b11-4796-a767-ab7402efbf87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 13:26:55.262418 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:55.262373 7010 patch_prober.go:29] interesting pod/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:26:55.262845 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:55.262449 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" podUID="03a68869-2b11-4796-a767-ab7402efbf87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 13:26:55.513731 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:55.513647 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe53a3fb-1f3d-463b-a165-713c55449df7" path="/var/lib/kubelet/pods/fe53a3fb-1f3d-463b-a165-713c55449df7/volumes" Oct 30 13:26:56.263122 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:56.263083 7010 patch_prober.go:29] interesting pod/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:26:56.263543 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:56.263166 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" podUID="03a68869-2b11-4796-a767-ab7402efbf87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 13:26:57.262774 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:57.262735 7010 patch_prober.go:29] interesting pod/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 13:26:57.262960 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:57.262796 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" podUID="03a68869-2b11-4796-a767-ab7402efbf87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 13:26:58.268112 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:58.268080 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:58.268605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:58.268591 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:58.269563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:58.269543 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:26:59.406142 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:26:59.406103 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:26:59.481146 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:59.481104 7010 generic.go:356] "Generic (PLEG): container finished" podID="f572a662-3ef2-4230-a1f5-6bf0ae106344" containerID="ac8582d7c6917ae1fcebc6059e0a100ceca867cdb88a9d6d32563ef0f6139bd0" exitCode=0 Oct 30 13:26:59.494187 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:59.494062 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" event={"ID":"f572a662-3ef2-4230-a1f5-6bf0ae106344","Type":"ContainerDied","Data":"ac8582d7c6917ae1fcebc6059e0a100ceca867cdb88a9d6d32563ef0f6139bd0"} Oct 30 13:26:59.551499 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:59.551448 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" Oct 30 13:26:59.652530 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:59.652455 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f572a662-3ef2-4230-a1f5-6bf0ae106344-kube-api-access-hw8tq\" (UniqueName: \"kubernetes.io/projected/f572a662-3ef2-4230-a1f5-6bf0ae106344-kube-api-access-hw8tq\") pod \"f572a662-3ef2-4230-a1f5-6bf0ae106344\" (UID: \"f572a662-3ef2-4230-a1f5-6bf0ae106344\") " Oct 30 13:26:59.652726 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:59.652665 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f572a662-3ef2-4230-a1f5-6bf0ae106344-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f572a662-3ef2-4230-a1f5-6bf0ae106344-plugins-config-volume\") pod \"f572a662-3ef2-4230-a1f5-6bf0ae106344\" (UID: \"f572a662-3ef2-4230-a1f5-6bf0ae106344\") " Oct 30 13:26:59.653026 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:59.653000 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f572a662-3ef2-4230-a1f5-6bf0ae106344-plugins-config-volume" pod "f572a662-3ef2-4230-a1f5-6bf0ae106344" (UID: "f572a662-3ef2-4230-a1f5-6bf0ae106344"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:26:59.653215 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:59.653195 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f572a662-3ef2-4230-a1f5-6bf0ae106344-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:26:59.654587 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:59.654566 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f572a662-3ef2-4230-a1f5-6bf0ae106344-kube-api-access-hw8tq" pod "f572a662-3ef2-4230-a1f5-6bf0ae106344" (UID: "f572a662-3ef2-4230-a1f5-6bf0ae106344"). InnerVolumeSpecName "kube-api-access-hw8tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:26:59.755106 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:26:59.755066 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw8tq\" (UniqueName: \"kubernetes.io/projected/f572a662-3ef2-4230-a1f5-6bf0ae106344-kube-api-access-hw8tq\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:00.489687 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:00.489657 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" Oct 30 13:27:00.490124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:00.489657 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z" event={"ID":"f572a662-3ef2-4230-a1f5-6bf0ae106344","Type":"ContainerDied","Data":"6532c8b2c74fe4c0d333dcc0539a9430041af2bd776d80bb6665b583724b563e"} Oct 30 13:27:00.490124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:00.489790 7010 scope.go:122] "RemoveContainer" containerID="ac8582d7c6917ae1fcebc6059e0a100ceca867cdb88a9d6d32563ef0f6139bd0" Oct 30 13:27:00.545744 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:00.545708 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z"] Oct 30 13:27:00.558310 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:00.558270 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b5a369-d5bf9ff67-z652z"] Oct 30 13:27:01.490388 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:01.490357 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f572a662-3ef2-4230-a1f5-6bf0ae106344" path="/var/lib/kubelet/pods/f572a662-3ef2-4230-a1f5-6bf0ae106344/volumes" Oct 30 13:27:04.142427 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.142344 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd"] Oct 30 13:27:04.142910 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.142653 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" podUID="3f0de2c5-3253-402e-99ec-70d8fb93a8e8" containerName="maas-api" containerID="cri-o://a2e09a2f4ed9cab397a5f512584888347584d56f17e30d1c2133162264239399" gracePeriod=30 Oct 30 13:27:04.224046 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.223995 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" podUID="3f0de2c5-3253-402e-99ec-70d8fb93a8e8" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.96:8443/health\": dial tcp 10.134.0.96:8443: connect: connection refused" Oct 30 13:27:04.541664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.541626 7010 generic.go:356] "Generic (PLEG): container finished" podID="3f0de2c5-3253-402e-99ec-70d8fb93a8e8" containerID="a2e09a2f4ed9cab397a5f512584888347584d56f17e30d1c2133162264239399" exitCode=0 Oct 30 13:27:04.541832 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.541699 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" event={"ID":"3f0de2c5-3253-402e-99ec-70d8fb93a8e8","Type":"ContainerDied","Data":"a2e09a2f4ed9cab397a5f512584888347584d56f17e30d1c2133162264239399"} Oct 30 13:27:04.541832 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.541751 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" event={"ID":"3f0de2c5-3253-402e-99ec-70d8fb93a8e8","Type":"ContainerDied","Data":"8c3d514dcf59a827f3488a8e8c64355bafde64fa9da828359807efd7dbc45c78"} Oct 30 13:27:04.541832 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.541773 7010 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c3d514dcf59a827f3488a8e8c64355bafde64fa9da828359807efd7dbc45c78" Oct 30 13:27:04.576511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.573976 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:27:04.654810 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.654148 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-openshift-service-ca\") pod \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\" (UID: \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\") " Oct 30 13:27:04.654810 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.654208 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-maas-api-tls\") pod \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\" (UID: \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\") " Oct 30 13:27:04.654810 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.654265 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-metrics-certs\") pod \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\" (UID: \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\") " Oct 30 13:27:04.654810 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.654368 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-kube-api-access-6j4rk\" (UniqueName: \"kubernetes.io/projected/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-kube-api-access-6j4rk\") pod \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\" (UID: \"3f0de2c5-3253-402e-99ec-70d8fb93a8e8\") " Oct 30 13:27:04.658094 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.658005 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-openshift-service-ca" pod "3f0de2c5-3253-402e-99ec-70d8fb93a8e8" (UID: "3f0de2c5-3253-402e-99ec-70d8fb93a8e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:04.670508 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.663322 7010 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-openshift-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:04.720529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.720101 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-maas-api-tls" pod "3f0de2c5-3253-402e-99ec-70d8fb93a8e8" (UID: "3f0de2c5-3253-402e-99ec-70d8fb93a8e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:27:04.741282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.739162 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-metrics-certs" pod "3f0de2c5-3253-402e-99ec-70d8fb93a8e8" (UID: "3f0de2c5-3253-402e-99ec-70d8fb93a8e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:27:04.747914 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.747868 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-kube-api-access-6j4rk" pod "3f0de2c5-3253-402e-99ec-70d8fb93a8e8" (UID: "3f0de2c5-3253-402e-99ec-70d8fb93a8e8"). InnerVolumeSpecName "kube-api-access-6j4rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:04.769685 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.769593 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j4rk\" (UniqueName: \"kubernetes.io/projected/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-kube-api-access-6j4rk\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:04.769895 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.769871 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-maas-api-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:04.769995 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:04.769982 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0de2c5-3253-402e-99ec-70d8fb93a8e8-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:05.549062 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:05.549031 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd" Oct 30 13:27:05.602764 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:05.602725 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd"] Oct 30 13:27:05.615688 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:05.615650 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b5a369-75cdb8ddd5-t4sbd"] Oct 30 13:27:07.490075 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:07.490036 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f0de2c5-3253-402e-99ec-70d8fb93a8e8" path="/var/lib/kubelet/pods/3f0de2c5-3253-402e-99ec-70d8fb93a8e8/volumes" Oct 30 13:27:23.022075 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:23.022034 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct"] Oct 30 13:27:23.025407 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:23.022276 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" podUID="7824b283-1848-4c7c-a66c-a53c57354c0d" containerName="istio-proxy" containerID="cri-o://a7f6ee046fe4bf16697f3ab832a9af3364035851899faac79628126c68598cbf" gracePeriod=30 Oct 30 13:27:23.035711 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:23.035676 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585"] Oct 30 13:27:23.035935 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:23.035907 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" podUID="db8b7959-d693-43bb-b380-1b1ae29776f2" containerName="istio-proxy" containerID="cri-o://67aeff2e357d26325df5cc3f35f9dcd58b7fedff85f962bf7d00fd6813eb3c49" gracePeriod=30 Oct 30 13:27:23.835096 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:23.835065 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:27:23.836935 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:23.836908 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:27:24.921888 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:24.921853 7010 scope.go:122] "RemoveContainer" containerID="a2e09a2f4ed9cab397a5f512584888347584d56f17e30d1c2133162264239399" Oct 30 13:27:25.385762 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:25.385247 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j"] Oct 30 13:27:25.386310 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:25.386271 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" podUID="51645cac-ccad-44c3-9928-d0283dc273ee" containerName="istio-proxy" containerID="cri-o://c3724b123a8783f7c38e26175f991dfe5786b82f0f2c5443a05a1e3d7166489b" gracePeriod=30 Oct 30 13:27:28.347957 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.347926 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:27:28.392323 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.392289 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:27:28.473749 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.473699 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7824b283-1848-4c7c-a66c-a53c57354c0d-kube-api-access-nnfkg\" (UniqueName: \"kubernetes.io/projected/7824b283-1848-4c7c-a66c-a53c57354c0d-kube-api-access-nnfkg\") pod \"7824b283-1848-4c7c-a66c-a53c57354c0d\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " Oct 30 13:27:28.473936 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.473766 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-data\") pod \"db8b7959-d693-43bb-b380-1b1ae29776f2\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " Oct 30 13:27:28.473936 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.473806 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db8b7959-d693-43bb-b380-1b1ae29776f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8b7959-d693-43bb-b380-1b1ae29776f2-istiod-ca-cert\") pod \"db8b7959-d693-43bb-b380-1b1ae29776f2\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " Oct 30 13:27:28.474101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474082 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-workload-socket\") pod \"7824b283-1848-4c7c-a66c-a53c57354c0d\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " Oct 30 13:27:28.474177 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474160 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-token\" (UniqueName: \"kubernetes.io/projected/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-token\") pod \"db8b7959-d693-43bb-b380-1b1ae29776f2\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " Oct 30 13:27:28.474240 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474216 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-podinfo\") pod \"7824b283-1848-4c7c-a66c-a53c57354c0d\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " Oct 30 13:27:28.474351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474334 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-envoy\") pod \"db8b7959-d693-43bb-b380-1b1ae29776f2\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " Oct 30 13:27:28.474424 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474368 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-credential-socket\") pod \"db8b7959-d693-43bb-b380-1b1ae29776f2\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " Oct 30 13:27:28.474424 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474368 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-workload-socket" pod "7824b283-1848-4c7c-a66c-a53c57354c0d" (UID: "7824b283-1848-4c7c-a66c-a53c57354c0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:28.474424 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474396 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-workload-certs\") pod \"db8b7959-d693-43bb-b380-1b1ae29776f2\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " Oct 30 13:27:28.474424 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474399 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db8b7959-d693-43bb-b380-1b1ae29776f2-istiod-ca-cert" pod "db8b7959-d693-43bb-b380-1b1ae29776f2" (UID: "db8b7959-d693-43bb-b380-1b1ae29776f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:28.474725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474443 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-envoy\") pod \"7824b283-1848-4c7c-a66c-a53c57354c0d\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " Oct 30 13:27:28.474725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474512 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db8b7959-d693-43bb-b380-1b1ae29776f2-kube-api-access-lrfdn\" (UniqueName: \"kubernetes.io/projected/db8b7959-d693-43bb-b380-1b1ae29776f2-kube-api-access-lrfdn\") pod \"db8b7959-d693-43bb-b380-1b1ae29776f2\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " Oct 30 13:27:28.474725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474545 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7824b283-1848-4c7c-a66c-a53c57354c0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7824b283-1848-4c7c-a66c-a53c57354c0d-istiod-ca-cert\") pod \"7824b283-1848-4c7c-a66c-a53c57354c0d\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " Oct 30 13:27:28.474725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474578 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-credential-socket\") pod \"7824b283-1848-4c7c-a66c-a53c57354c0d\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " Oct 30 13:27:28.474725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474584 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-data" pod "db8b7959-d693-43bb-b380-1b1ae29776f2" (UID: "db8b7959-d693-43bb-b380-1b1ae29776f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:28.474725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474614 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-data\") pod \"7824b283-1848-4c7c-a66c-a53c57354c0d\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " Oct 30 13:27:28.474725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474652 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-workload-socket\") pod \"db8b7959-d693-43bb-b380-1b1ae29776f2\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " Oct 30 13:27:28.474725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474697 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-podinfo\") pod \"db8b7959-d693-43bb-b380-1b1ae29776f2\" (UID: \"db8b7959-d693-43bb-b380-1b1ae29776f2\") " Oct 30 13:27:28.475179 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474774 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-workload-certs\") pod \"7824b283-1848-4c7c-a66c-a53c57354c0d\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " Oct 30 13:27:28.475179 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.474825 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-token\" (UniqueName: \"kubernetes.io/projected/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-token\") pod \"7824b283-1848-4c7c-a66c-a53c57354c0d\" (UID: \"7824b283-1848-4c7c-a66c-a53c57354c0d\") " Oct 30 13:27:28.475292 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.475209 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-workload-certs" pod "db8b7959-d693-43bb-b380-1b1ae29776f2" (UID: "db8b7959-d693-43bb-b380-1b1ae29776f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:28.475612 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.475588 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.475704 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.475615 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.475973 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.475629 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8b7959-d693-43bb-b380-1b1ae29776f2-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.475973 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.475964 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.476114 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.475791 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-credential-socket" pod "db8b7959-d693-43bb-b380-1b1ae29776f2" (UID: "db8b7959-d693-43bb-b380-1b1ae29776f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:28.476114 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.475967 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-credential-socket" pod "7824b283-1848-4c7c-a66c-a53c57354c0d" (UID: "7824b283-1848-4c7c-a66c-a53c57354c0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:28.476232 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.476171 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-workload-socket" pod "db8b7959-d693-43bb-b380-1b1ae29776f2" (UID: "db8b7959-d693-43bb-b380-1b1ae29776f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:28.476477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.476453 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-data" pod "7824b283-1848-4c7c-a66c-a53c57354c0d" (UID: "7824b283-1848-4c7c-a66c-a53c57354c0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:28.477256 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.477229 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-workload-certs" pod "7824b283-1848-4c7c-a66c-a53c57354c0d" (UID: "7824b283-1848-4c7c-a66c-a53c57354c0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:28.477611 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.477584 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7824b283-1848-4c7c-a66c-a53c57354c0d-istiod-ca-cert" pod "7824b283-1848-4c7c-a66c-a53c57354c0d" (UID: "7824b283-1848-4c7c-a66c-a53c57354c0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:28.479041 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.479018 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-envoy" pod "7824b283-1848-4c7c-a66c-a53c57354c0d" (UID: "7824b283-1848-4c7c-a66c-a53c57354c0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:28.479181 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.479152 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-podinfo" pod "7824b283-1848-4c7c-a66c-a53c57354c0d" (UID: "7824b283-1848-4c7c-a66c-a53c57354c0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:27:28.479271 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.479178 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-token" pod "7824b283-1848-4c7c-a66c-a53c57354c0d" (UID: "7824b283-1848-4c7c-a66c-a53c57354c0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:28.479271 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.479242 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-envoy" pod "db8b7959-d693-43bb-b380-1b1ae29776f2" (UID: "db8b7959-d693-43bb-b380-1b1ae29776f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:28.479551 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.479521 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-token" pod "db8b7959-d693-43bb-b380-1b1ae29776f2" (UID: "db8b7959-d693-43bb-b380-1b1ae29776f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:28.479991 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.479970 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7824b283-1848-4c7c-a66c-a53c57354c0d-kube-api-access-nnfkg" pod "7824b283-1848-4c7c-a66c-a53c57354c0d" (UID: "7824b283-1848-4c7c-a66c-a53c57354c0d"). InnerVolumeSpecName "kube-api-access-nnfkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:28.481194 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.481168 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-podinfo" pod "db8b7959-d693-43bb-b380-1b1ae29776f2" (UID: "db8b7959-d693-43bb-b380-1b1ae29776f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:27:28.481294 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.481225 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8b7959-d693-43bb-b380-1b1ae29776f2-kube-api-access-lrfdn" pod "db8b7959-d693-43bb-b380-1b1ae29776f2" (UID: "db8b7959-d693-43bb-b380-1b1ae29776f2"). InnerVolumeSpecName "kube-api-access-lrfdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:28.577185 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.577081 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.577185 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.577118 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrfdn\" (UniqueName: \"kubernetes.io/projected/db8b7959-d693-43bb-b380-1b1ae29776f2-kube-api-access-lrfdn\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.577185 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.577128 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7824b283-1848-4c7c-a66c-a53c57354c0d-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.577185 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.577138 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.577185 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.577146 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.577185 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.577159 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.577185 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.577170 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.577185 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.577178 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7824b283-1848-4c7c-a66c-a53c57354c0d-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.577185 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.577185 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.577185 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.577193 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnfkg\" (UniqueName: \"kubernetes.io/projected/7824b283-1848-4c7c-a66c-a53c57354c0d-kube-api-access-nnfkg\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.577863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.577200 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.577863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.577209 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7824b283-1848-4c7c-a66c-a53c57354c0d-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.577863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.577218 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.577863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.577227 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8b7959-d693-43bb-b380-1b1ae29776f2-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:28.747124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.747089 7010 generic.go:356] "Generic (PLEG): container finished" podID="7824b283-1848-4c7c-a66c-a53c57354c0d" containerID="a7f6ee046fe4bf16697f3ab832a9af3364035851899faac79628126c68598cbf" exitCode=0 Oct 30 13:27:28.747335 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.747148 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" Oct 30 13:27:28.747335 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.747164 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" event={"ID":"7824b283-1848-4c7c-a66c-a53c57354c0d","Type":"ContainerDied","Data":"a7f6ee046fe4bf16697f3ab832a9af3364035851899faac79628126c68598cbf"} Oct 30 13:27:28.747335 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.747219 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct" event={"ID":"7824b283-1848-4c7c-a66c-a53c57354c0d","Type":"ContainerDied","Data":"6efca58ae19c07e0272ecab240d0c0677a3d50207a3da83142df6451fb200f48"} Oct 30 13:27:28.747335 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.747245 7010 scope.go:122] "RemoveContainer" containerID="a7f6ee046fe4bf16697f3ab832a9af3364035851899faac79628126c68598cbf" Oct 30 13:27:28.749044 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.749015 7010 generic.go:356] "Generic (PLEG): container finished" podID="db8b7959-d693-43bb-b380-1b1ae29776f2" containerID="67aeff2e357d26325df5cc3f35f9dcd58b7fedff85f962bf7d00fd6813eb3c49" exitCode=0 Oct 30 13:27:28.749176 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.749052 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" event={"ID":"db8b7959-d693-43bb-b380-1b1ae29776f2","Type":"ContainerDied","Data":"67aeff2e357d26325df5cc3f35f9dcd58b7fedff85f962bf7d00fd6813eb3c49"} Oct 30 13:27:28.749176 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.749081 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" Oct 30 13:27:28.749176 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.749098 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585" event={"ID":"db8b7959-d693-43bb-b380-1b1ae29776f2","Type":"ContainerDied","Data":"22d90d7dae28f61ce65bf4ef7598dd457acf8752058860942b369038aca87910"} Oct 30 13:27:28.761719 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.761691 7010 scope.go:122] "RemoveContainer" containerID="a7f6ee046fe4bf16697f3ab832a9af3364035851899faac79628126c68598cbf" Oct 30 13:27:28.762396 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:27:28.762364 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f6ee046fe4bf16697f3ab832a9af3364035851899faac79628126c68598cbf\": container with ID starting with a7f6ee046fe4bf16697f3ab832a9af3364035851899faac79628126c68598cbf not found: ID does not exist" containerID="a7f6ee046fe4bf16697f3ab832a9af3364035851899faac79628126c68598cbf" Oct 30 13:27:28.762531 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.762409 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f6ee046fe4bf16697f3ab832a9af3364035851899faac79628126c68598cbf"} err="failed to get container status \"a7f6ee046fe4bf16697f3ab832a9af3364035851899faac79628126c68598cbf\": rpc error: code = NotFound desc = could not find container \"a7f6ee046fe4bf16697f3ab832a9af3364035851899faac79628126c68598cbf\": container with ID starting with a7f6ee046fe4bf16697f3ab832a9af3364035851899faac79628126c68598cbf not found: ID does not exist" Oct 30 13:27:28.762531 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.762433 7010 scope.go:122] "RemoveContainer" containerID="67aeff2e357d26325df5cc3f35f9dcd58b7fedff85f962bf7d00fd6813eb3c49" Oct 30 13:27:28.773944 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.773920 7010 scope.go:122] "RemoveContainer" containerID="67aeff2e357d26325df5cc3f35f9dcd58b7fedff85f962bf7d00fd6813eb3c49" Oct 30 13:27:28.774297 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:27:28.774279 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67aeff2e357d26325df5cc3f35f9dcd58b7fedff85f962bf7d00fd6813eb3c49\": container with ID starting with 67aeff2e357d26325df5cc3f35f9dcd58b7fedff85f962bf7d00fd6813eb3c49 not found: ID does not exist" containerID="67aeff2e357d26325df5cc3f35f9dcd58b7fedff85f962bf7d00fd6813eb3c49" Oct 30 13:27:28.774402 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.774307 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67aeff2e357d26325df5cc3f35f9dcd58b7fedff85f962bf7d00fd6813eb3c49"} err="failed to get container status \"67aeff2e357d26325df5cc3f35f9dcd58b7fedff85f962bf7d00fd6813eb3c49\": rpc error: code = NotFound desc = could not find container \"67aeff2e357d26325df5cc3f35f9dcd58b7fedff85f962bf7d00fd6813eb3c49\": container with ID starting with 67aeff2e357d26325df5cc3f35f9dcd58b7fedff85f962bf7d00fd6813eb3c49 not found: ID does not exist" Oct 30 13:27:28.802775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.802730 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct"] Oct 30 13:27:28.815262 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.815216 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-vzfct"] Oct 30 13:27:28.850582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.844353 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585"] Oct 30 13:27:28.859625 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:28.859559 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-jp585"] Oct 30 13:27:29.402590 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:27:29.402547 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:27:29.490235 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:29.490194 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7824b283-1848-4c7c-a66c-a53c57354c0d" path="/var/lib/kubelet/pods/7824b283-1848-4c7c-a66c-a53c57354c0d/volumes" Oct 30 13:27:29.490764 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:29.490748 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db8b7959-d693-43bb-b380-1b1ae29776f2" path="/var/lib/kubelet/pods/db8b7959-d693-43bb-b380-1b1ae29776f2/volumes" Oct 30 13:27:30.697141 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.697111 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:27:30.769453 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.769417 7010 generic.go:356] "Generic (PLEG): container finished" podID="51645cac-ccad-44c3-9928-d0283dc273ee" containerID="c3724b123a8783f7c38e26175f991dfe5786b82f0f2c5443a05a1e3d7166489b" exitCode=0 Oct 30 13:27:30.769682 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.769506 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" Oct 30 13:27:30.769682 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.769522 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" event={"ID":"51645cac-ccad-44c3-9928-d0283dc273ee","Type":"ContainerDied","Data":"c3724b123a8783f7c38e26175f991dfe5786b82f0f2c5443a05a1e3d7166489b"} Oct 30 13:27:30.769682 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.769578 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j" event={"ID":"51645cac-ccad-44c3-9928-d0283dc273ee","Type":"ContainerDied","Data":"15a87e568e64943d4d1be45233977515d1c7be57b33528147d95d62a48834e02"} Oct 30 13:27:30.769682 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.769605 7010 scope.go:122] "RemoveContainer" containerID="c3724b123a8783f7c38e26175f991dfe5786b82f0f2c5443a05a1e3d7166489b" Oct 30 13:27:30.787251 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.787228 7010 scope.go:122] "RemoveContainer" containerID="203d983a17ecd15c8e8f0f201bc685f37df111f4713ea8702c50ec0cd7e3cfc9" Oct 30 13:27:30.798792 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.798763 7010 scope.go:122] "RemoveContainer" containerID="c3724b123a8783f7c38e26175f991dfe5786b82f0f2c5443a05a1e3d7166489b" Oct 30 13:27:30.799163 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:27:30.799140 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3724b123a8783f7c38e26175f991dfe5786b82f0f2c5443a05a1e3d7166489b\": container with ID starting with c3724b123a8783f7c38e26175f991dfe5786b82f0f2c5443a05a1e3d7166489b not found: ID does not exist" containerID="c3724b123a8783f7c38e26175f991dfe5786b82f0f2c5443a05a1e3d7166489b" Oct 30 13:27:30.799267 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.799173 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3724b123a8783f7c38e26175f991dfe5786b82f0f2c5443a05a1e3d7166489b"} err="failed to get container status \"c3724b123a8783f7c38e26175f991dfe5786b82f0f2c5443a05a1e3d7166489b\": rpc error: code = NotFound desc = could not find container \"c3724b123a8783f7c38e26175f991dfe5786b82f0f2c5443a05a1e3d7166489b\": container with ID starting with c3724b123a8783f7c38e26175f991dfe5786b82f0f2c5443a05a1e3d7166489b not found: ID does not exist" Oct 30 13:27:30.799267 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.799190 7010 scope.go:122] "RemoveContainer" containerID="203d983a17ecd15c8e8f0f201bc685f37df111f4713ea8702c50ec0cd7e3cfc9" Oct 30 13:27:30.799540 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:27:30.799480 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"203d983a17ecd15c8e8f0f201bc685f37df111f4713ea8702c50ec0cd7e3cfc9\": container with ID starting with 203d983a17ecd15c8e8f0f201bc685f37df111f4713ea8702c50ec0cd7e3cfc9 not found: ID does not exist" containerID="203d983a17ecd15c8e8f0f201bc685f37df111f4713ea8702c50ec0cd7e3cfc9" Oct 30 13:27:30.799634 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.799546 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203d983a17ecd15c8e8f0f201bc685f37df111f4713ea8702c50ec0cd7e3cfc9"} err="failed to get container status \"203d983a17ecd15c8e8f0f201bc685f37df111f4713ea8702c50ec0cd7e3cfc9\": rpc error: code = NotFound desc = could not find container \"203d983a17ecd15c8e8f0f201bc685f37df111f4713ea8702c50ec0cd7e3cfc9\": container with ID starting with 203d983a17ecd15c8e8f0f201bc685f37df111f4713ea8702c50ec0cd7e3cfc9 not found: ID does not exist" Oct 30 13:27:30.815656 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.815619 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-workload-socket\") pod \"51645cac-ccad-44c3-9928-d0283dc273ee\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " Oct 30 13:27:30.815791 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.815675 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-workload-certs\") pod \"51645cac-ccad-44c3-9928-d0283dc273ee\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " Oct 30 13:27:30.815791 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.815726 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-istio-data\") pod \"51645cac-ccad-44c3-9928-d0283dc273ee\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " Oct 30 13:27:30.815910 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.815795 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51645cac-ccad-44c3-9928-d0283dc273ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51645cac-ccad-44c3-9928-d0283dc273ee-istio-podinfo\") pod \"51645cac-ccad-44c3-9928-d0283dc273ee\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " Oct 30 13:27:30.815910 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.815838 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51645cac-ccad-44c3-9928-d0283dc273ee-kube-api-access-6z2wt\" (UniqueName: \"kubernetes.io/projected/51645cac-ccad-44c3-9928-d0283dc273ee-kube-api-access-6z2wt\") pod \"51645cac-ccad-44c3-9928-d0283dc273ee\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " Oct 30 13:27:30.815910 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.815881 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51645cac-ccad-44c3-9928-d0283dc273ee-istio-token\" (UniqueName: \"kubernetes.io/projected/51645cac-ccad-44c3-9928-d0283dc273ee-istio-token\") pod \"51645cac-ccad-44c3-9928-d0283dc273ee\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " Oct 30 13:27:30.816071 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.815929 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-workload-certs" pod "51645cac-ccad-44c3-9928-d0283dc273ee" (UID: "51645cac-ccad-44c3-9928-d0283dc273ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:30.816071 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.815940 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-workload-socket" pod "51645cac-ccad-44c3-9928-d0283dc273ee" (UID: "51645cac-ccad-44c3-9928-d0283dc273ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:30.816071 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.815986 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-credential-socket\") pod \"51645cac-ccad-44c3-9928-d0283dc273ee\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " Oct 30 13:27:30.816229 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.816061 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-istio-data" pod "51645cac-ccad-44c3-9928-d0283dc273ee" (UID: "51645cac-ccad-44c3-9928-d0283dc273ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:30.816229 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.816080 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-istio-envoy\") pod \"51645cac-ccad-44c3-9928-d0283dc273ee\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " Oct 30 13:27:30.816229 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.816115 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51645cac-ccad-44c3-9928-d0283dc273ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51645cac-ccad-44c3-9928-d0283dc273ee-istiod-ca-cert\") pod \"51645cac-ccad-44c3-9928-d0283dc273ee\" (UID: \"51645cac-ccad-44c3-9928-d0283dc273ee\") " Oct 30 13:27:30.816394 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.816298 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-credential-socket" pod "51645cac-ccad-44c3-9928-d0283dc273ee" (UID: "51645cac-ccad-44c3-9928-d0283dc273ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:30.816654 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.816631 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51645cac-ccad-44c3-9928-d0283dc273ee-istiod-ca-cert" pod "51645cac-ccad-44c3-9928-d0283dc273ee" (UID: "51645cac-ccad-44c3-9928-d0283dc273ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:30.816881 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.816789 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51645cac-ccad-44c3-9928-d0283dc273ee-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:30.816881 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.816817 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:30.816881 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.816832 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:30.816881 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.816845 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:30.816881 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.816860 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:30.818380 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.818353 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51645cac-ccad-44c3-9928-d0283dc273ee-kube-api-access-6z2wt" pod "51645cac-ccad-44c3-9928-d0283dc273ee" (UID: "51645cac-ccad-44c3-9928-d0283dc273ee"). InnerVolumeSpecName "kube-api-access-6z2wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:30.818547 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.818517 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51645cac-ccad-44c3-9928-d0283dc273ee-istio-podinfo" pod "51645cac-ccad-44c3-9928-d0283dc273ee" (UID: "51645cac-ccad-44c3-9928-d0283dc273ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:27:30.818721 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.818698 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51645cac-ccad-44c3-9928-d0283dc273ee-istio-token" pod "51645cac-ccad-44c3-9928-d0283dc273ee" (UID: "51645cac-ccad-44c3-9928-d0283dc273ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:30.819583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.819562 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-istio-envoy" pod "51645cac-ccad-44c3-9928-d0283dc273ee" (UID: "51645cac-ccad-44c3-9928-d0283dc273ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:27:30.918074 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.917994 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z2wt\" (UniqueName: \"kubernetes.io/projected/51645cac-ccad-44c3-9928-d0283dc273ee-kube-api-access-6z2wt\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:30.918074 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.918023 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51645cac-ccad-44c3-9928-d0283dc273ee-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:30.918074 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.918032 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51645cac-ccad-44c3-9928-d0283dc273ee-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:30.918074 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:30.918041 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51645cac-ccad-44c3-9928-d0283dc273ee-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:31.123772 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:31.123735 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j"] Oct 30 13:27:31.128348 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:31.128311 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5dd8cc-openshift-default-67d4565867-tvp5j"] Oct 30 13:27:31.491221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:31.491179 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51645cac-ccad-44c3-9928-d0283dc273ee" path="/var/lib/kubelet/pods/51645cac-ccad-44c3-9928-d0283dc273ee/volumes" Oct 30 13:27:52.518883 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:52.518770 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4"] Oct 30 13:27:57.520121 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:57.520077 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" podUID="5d3233ae-f903-495a-aa4c-1e8fe88efa5b" containerName="payload-processing" containerID="cri-o://da03b500e73b5b9495fb9e221b8c71311828b2b1839bc2b16c66786d91858f87" gracePeriod=25 Oct 30 13:27:57.802482 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:57.802452 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" Oct 30 13:27:57.918208 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:57.918175 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d3233ae-f903-495a-aa4c-1e8fe88efa5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3233ae-f903-495a-aa4c-1e8fe88efa5b-plugins-config-volume\") pod \"5d3233ae-f903-495a-aa4c-1e8fe88efa5b\" (UID: \"5d3233ae-f903-495a-aa4c-1e8fe88efa5b\") " Oct 30 13:27:57.918208 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:57.918226 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3233ae-f903-495a-aa4c-1e8fe88efa5b-kube-api-access-qsw27\" (UniqueName: \"kubernetes.io/projected/5d3233ae-f903-495a-aa4c-1e8fe88efa5b-kube-api-access-qsw27\") pod \"5d3233ae-f903-495a-aa4c-1e8fe88efa5b\" (UID: \"5d3233ae-f903-495a-aa4c-1e8fe88efa5b\") " Oct 30 13:27:57.918716 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:57.918693 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3233ae-f903-495a-aa4c-1e8fe88efa5b-plugins-config-volume" pod "5d3233ae-f903-495a-aa4c-1e8fe88efa5b" (UID: "5d3233ae-f903-495a-aa4c-1e8fe88efa5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:27:57.931610 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:57.920843 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3233ae-f903-495a-aa4c-1e8fe88efa5b-kube-api-access-qsw27" pod "5d3233ae-f903-495a-aa4c-1e8fe88efa5b" (UID: "5d3233ae-f903-495a-aa4c-1e8fe88efa5b"). InnerVolumeSpecName "kube-api-access-qsw27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:27:57.968395 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:57.968355 7010 generic.go:356] "Generic (PLEG): container finished" podID="5d3233ae-f903-495a-aa4c-1e8fe88efa5b" containerID="da03b500e73b5b9495fb9e221b8c71311828b2b1839bc2b16c66786d91858f87" exitCode=0 Oct 30 13:27:57.968607 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:57.968405 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" Oct 30 13:27:57.968607 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:57.968432 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" event={"ID":"5d3233ae-f903-495a-aa4c-1e8fe88efa5b","Type":"ContainerDied","Data":"da03b500e73b5b9495fb9e221b8c71311828b2b1839bc2b16c66786d91858f87"} Oct 30 13:27:57.968607 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:57.968501 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4" event={"ID":"5d3233ae-f903-495a-aa4c-1e8fe88efa5b","Type":"ContainerDied","Data":"ba4efd88b1a14a4873365938ad3cabde6311e0156ba89fdd43c8900c7f9b41dc"} Oct 30 13:27:57.968607 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:57.968521 7010 scope.go:122] "RemoveContainer" containerID="da03b500e73b5b9495fb9e221b8c71311828b2b1839bc2b16c66786d91858f87" Oct 30 13:27:57.979354 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:57.979327 7010 scope.go:122] "RemoveContainer" containerID="da03b500e73b5b9495fb9e221b8c71311828b2b1839bc2b16c66786d91858f87" Oct 30 13:27:57.979729 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:27:57.979702 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da03b500e73b5b9495fb9e221b8c71311828b2b1839bc2b16c66786d91858f87\": container with ID starting with da03b500e73b5b9495fb9e221b8c71311828b2b1839bc2b16c66786d91858f87 not found: ID does not exist" containerID="da03b500e73b5b9495fb9e221b8c71311828b2b1839bc2b16c66786d91858f87" Oct 30 13:27:57.979808 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:57.979737 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da03b500e73b5b9495fb9e221b8c71311828b2b1839bc2b16c66786d91858f87"} err="failed to get container status \"da03b500e73b5b9495fb9e221b8c71311828b2b1839bc2b16c66786d91858f87\": rpc error: code = NotFound desc = could not find container \"da03b500e73b5b9495fb9e221b8c71311828b2b1839bc2b16c66786d91858f87\": container with ID starting with da03b500e73b5b9495fb9e221b8c71311828b2b1839bc2b16c66786d91858f87 not found: ID does not exist" Oct 30 13:27:58.019850 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:58.019812 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d3233ae-f903-495a-aa4c-1e8fe88efa5b-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:58.019850 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:58.019842 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsw27\" (UniqueName: \"kubernetes.io/projected/5d3233ae-f903-495a-aa4c-1e8fe88efa5b-kube-api-access-qsw27\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:27:58.023333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:58.023265 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4"] Oct 30 13:27:58.027355 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:58.027329 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09e4c0-69b5d5d764-27gx4"] Oct 30 13:27:59.400828 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:27:59.400785 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:27:59.489053 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:27:59.489023 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d3233ae-f903-495a-aa4c-1e8fe88efa5b" path="/var/lib/kubelet/pods/5d3233ae-f903-495a-aa4c-1e8fe88efa5b/volumes" Oct 30 13:28:04.747667 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:04.747612 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g"] Oct 30 13:28:19.774347 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:19.774303 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" podUID="8422efa7-cca3-4534-983e-108c338e1c16" containerName="main" containerID="cri-o://3bd9a7ff159760a7a583db1de24f0ea167fb288e4dc223e47ea5fd566eb88fd5" gracePeriod=45 Oct 30 13:28:20.056038 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.056007 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:28:20.133753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.133705 7010 generic.go:356] "Generic (PLEG): container finished" podID="8422efa7-cca3-4534-983e-108c338e1c16" containerID="3bd9a7ff159760a7a583db1de24f0ea167fb288e4dc223e47ea5fd566eb88fd5" exitCode=0 Oct 30 13:28:20.133955 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.133763 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" Oct 30 13:28:20.133955 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.133799 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" event={"ID":"8422efa7-cca3-4534-983e-108c338e1c16","Type":"ContainerDied","Data":"3bd9a7ff159760a7a583db1de24f0ea167fb288e4dc223e47ea5fd566eb88fd5"} Oct 30 13:28:20.133955 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.133852 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g" event={"ID":"8422efa7-cca3-4534-983e-108c338e1c16","Type":"ContainerDied","Data":"02da4cf19d7e62c96eef20d24eea72187d3e99df68101459ee61f3e789634d0b"} Oct 30 13:28:20.133955 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.133878 7010 scope.go:122] "RemoveContainer" containerID="3bd9a7ff159760a7a583db1de24f0ea167fb288e4dc223e47ea5fd566eb88fd5" Oct 30 13:28:20.145920 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.145903 7010 scope.go:122] "RemoveContainer" containerID="3bd9a7ff159760a7a583db1de24f0ea167fb288e4dc223e47ea5fd566eb88fd5" Oct 30 13:28:20.146244 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:28:20.146220 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bd9a7ff159760a7a583db1de24f0ea167fb288e4dc223e47ea5fd566eb88fd5\": container with ID starting with 3bd9a7ff159760a7a583db1de24f0ea167fb288e4dc223e47ea5fd566eb88fd5 not found: ID does not exist" containerID="3bd9a7ff159760a7a583db1de24f0ea167fb288e4dc223e47ea5fd566eb88fd5" Oct 30 13:28:20.146313 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.146254 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bd9a7ff159760a7a583db1de24f0ea167fb288e4dc223e47ea5fd566eb88fd5"} err="failed to get container status \"3bd9a7ff159760a7a583db1de24f0ea167fb288e4dc223e47ea5fd566eb88fd5\": rpc error: code = NotFound desc = could not find container \"3bd9a7ff159760a7a583db1de24f0ea167fb288e4dc223e47ea5fd566eb88fd5\": container with ID starting with 3bd9a7ff159760a7a583db1de24f0ea167fb288e4dc223e47ea5fd566eb88fd5 not found: ID does not exist" Oct 30 13:28:20.153881 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.153854 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8422efa7-cca3-4534-983e-108c338e1c16-kube-api-access-w868v\" (UniqueName: \"kubernetes.io/projected/8422efa7-cca3-4534-983e-108c338e1c16-kube-api-access-w868v\") pod \"8422efa7-cca3-4534-983e-108c338e1c16\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " Oct 30 13:28:20.154019 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.153913 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8422efa7-cca3-4534-983e-108c338e1c16-tls-certs\" (UniqueName: \"kubernetes.io/secret/8422efa7-cca3-4534-983e-108c338e1c16-tls-certs\") pod \"8422efa7-cca3-4534-983e-108c338e1c16\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " Oct 30 13:28:20.154019 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.153973 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-home\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-home\") pod \"8422efa7-cca3-4534-983e-108c338e1c16\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " Oct 30 13:28:20.154128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.154044 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-dshm\") pod \"8422efa7-cca3-4534-983e-108c338e1c16\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " Oct 30 13:28:20.154128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.154099 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-model-cache\") pod \"8422efa7-cca3-4534-983e-108c338e1c16\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " Oct 30 13:28:20.154263 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.154235 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-tmp-dir\") pod \"8422efa7-cca3-4534-983e-108c338e1c16\" (UID: \"8422efa7-cca3-4534-983e-108c338e1c16\") " Oct 30 13:28:20.154319 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.154264 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-home" pod "8422efa7-cca3-4534-983e-108c338e1c16" (UID: "8422efa7-cca3-4534-983e-108c338e1c16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:20.154484 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.154433 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-model-cache" pod "8422efa7-cca3-4534-983e-108c338e1c16" (UID: "8422efa7-cca3-4534-983e-108c338e1c16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:20.154638 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.154538 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-tmp-dir" pod "8422efa7-cca3-4534-983e-108c338e1c16" (UID: "8422efa7-cca3-4534-983e-108c338e1c16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:20.154984 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.154965 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:20.155025 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.154992 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:20.155025 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.155010 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:20.156366 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.156339 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-dshm" pod "8422efa7-cca3-4534-983e-108c338e1c16" (UID: "8422efa7-cca3-4534-983e-108c338e1c16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:20.156464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.156344 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8422efa7-cca3-4534-983e-108c338e1c16-tls-certs" pod "8422efa7-cca3-4534-983e-108c338e1c16" (UID: "8422efa7-cca3-4534-983e-108c338e1c16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:28:20.156535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.156461 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8422efa7-cca3-4534-983e-108c338e1c16-kube-api-access-w868v" pod "8422efa7-cca3-4534-983e-108c338e1c16" (UID: "8422efa7-cca3-4534-983e-108c338e1c16"). InnerVolumeSpecName "kube-api-access-w868v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:20.256235 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.256194 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w868v\" (UniqueName: \"kubernetes.io/projected/8422efa7-cca3-4534-983e-108c338e1c16-kube-api-access-w868v\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:20.256235 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.256227 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8422efa7-cca3-4534-983e-108c338e1c16-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:20.256235 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.256241 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8422efa7-cca3-4534-983e-108c338e1c16-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:20.481934 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.481906 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g"] Oct 30 13:28:20.486887 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:20.486861 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-47628d/test-model-47628d-kserve-765979b8c7-mrn4g"] Oct 30 13:28:21.489764 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:21.489731 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8422efa7-cca3-4534-983e-108c338e1c16" path="/var/lib/kubelet/pods/8422efa7-cca3-4534-983e-108c338e1c16/volumes" Oct 30 13:28:22.330094 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:22.330065 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz"] Oct 30 13:28:29.402191 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:28:29.402149 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:28:33.407275 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.407007 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd"] Oct 30 13:28:33.408108 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408011 7010 cpu_manager.go:414] "containerMap: removing container" podUID="5d3233ae-f903-495a-aa4c-1e8fe88efa5b" containerName="payload-processing" Oct 30 13:28:33.408108 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408039 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3233ae-f903-495a-aa4c-1e8fe88efa5b" containerName="payload-processing" Oct 30 13:28:33.408108 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408059 7010 cpu_manager.go:414] "containerMap: removing container" podUID="3f0de2c5-3253-402e-99ec-70d8fb93a8e8" containerName="maas-api" Oct 30 13:28:33.408108 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408067 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f0de2c5-3253-402e-99ec-70d8fb93a8e8" containerName="maas-api" Oct 30 13:28:33.408108 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408099 7010 cpu_manager.go:414] "containerMap: removing container" podUID="51645cac-ccad-44c3-9928-d0283dc273ee" containerName="istio-proxy" Oct 30 13:28:33.408108 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408107 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51645cac-ccad-44c3-9928-d0283dc273ee" containerName="istio-proxy" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408124 7010 cpu_manager.go:414] "containerMap: removing container" podUID="db8b7959-d693-43bb-b380-1b1ae29776f2" containerName="istio-proxy" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408131 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db8b7959-d693-43bb-b380-1b1ae29776f2" containerName="istio-proxy" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408141 7010 cpu_manager.go:414] "containerMap: removing container" podUID="f572a662-3ef2-4230-a1f5-6bf0ae106344" containerName="payload-pre-processing" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408148 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f572a662-3ef2-4230-a1f5-6bf0ae106344" containerName="payload-pre-processing" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408174 7010 cpu_manager.go:414] "containerMap: removing container" podUID="8422efa7-cca3-4534-983e-108c338e1c16" containerName="main" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408181 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8422efa7-cca3-4534-983e-108c338e1c16" containerName="main" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408202 7010 cpu_manager.go:414] "containerMap: removing container" podUID="51645cac-ccad-44c3-9928-d0283dc273ee" containerName="istio-proxy" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408211 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51645cac-ccad-44c3-9928-d0283dc273ee" containerName="istio-proxy" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408238 7010 cpu_manager.go:414] "containerMap: removing container" podUID="7824b283-1848-4c7c-a66c-a53c57354c0d" containerName="istio-proxy" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408245 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7824b283-1848-4c7c-a66c-a53c57354c0d" containerName="istio-proxy" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408258 7010 cpu_manager.go:414] "containerMap: removing container" podUID="fe53a3fb-1f3d-463b-a165-713c55449df7" containerName="main" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408266 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe53a3fb-1f3d-463b-a165-713c55449df7" containerName="main" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408410 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="7824b283-1848-4c7c-a66c-a53c57354c0d" containerName="istio-proxy" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408424 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe53a3fb-1f3d-463b-a165-713c55449df7" containerName="main" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408435 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3233ae-f903-495a-aa4c-1e8fe88efa5b" containerName="payload-processing" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408444 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f0de2c5-3253-402e-99ec-70d8fb93a8e8" containerName="maas-api" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408456 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="51645cac-ccad-44c3-9928-d0283dc273ee" containerName="istio-proxy" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408466 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="db8b7959-d693-43bb-b380-1b1ae29776f2" containerName="istio-proxy" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408475 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="f572a662-3ef2-4230-a1f5-6bf0ae106344" containerName="payload-pre-processing" Oct 30 13:28:33.408588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.408505 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="8422efa7-cca3-4534-983e-108c338e1c16" containerName="main" Oct 30 13:28:33.414797 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.414771 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.491167 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.491122 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-data\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.491369 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.491197 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-workload-socket\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.491369 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.491292 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-workload-certs\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.491369 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.491326 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-token\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.491616 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.491382 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-envoy\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.491616 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.491461 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-credential-socket\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.491616 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.491558 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qbrm\" (UniqueName: \"kubernetes.io/projected/415237a2-07c0-481d-bb7b-f818c83e92ce-kube-api-access-7qbrm\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.491784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.491677 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-podinfo\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.491784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.491722 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415237a2-07c0-481d-bb7b-f818c83e92ce-istiod-ca-cert\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.497976 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.497933 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-9cbcbc-openshift-default-dockercfg-hkxps\"" Oct 30 13:28:33.510853 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.510820 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd"] Oct 30 13:28:33.593597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.593554 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qbrm\" (UniqueName: \"kubernetes.io/projected/415237a2-07c0-481d-bb7b-f818c83e92ce-kube-api-access-7qbrm\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.593871 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.593841 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-podinfo\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.593993 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.593947 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415237a2-07c0-481d-bb7b-f818c83e92ce-istiod-ca-cert\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.594534 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.594120 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-data\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.594534 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.594278 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-workload-socket\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.594534 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.594457 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-data\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.594777 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.594592 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-workload-certs\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.594777 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.594641 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-token\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.594777 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.594685 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-workload-socket\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.594777 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.594740 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-envoy\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.594777 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.594759 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415237a2-07c0-481d-bb7b-f818c83e92ce-istiod-ca-cert\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.595063 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.594887 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-credential-socket\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.595414 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.595164 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-credential-socket\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.595414 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.595319 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-workload-certs\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.596963 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.596941 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-podinfo\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.597257 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.597234 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-envoy\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.619759 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.619716 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-token\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.619931 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.619778 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qbrm\" (UniqueName: \"kubernetes.io/projected/415237a2-07c0-481d-bb7b-f818c83e92ce-kube-api-access-7qbrm\") pod \"e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:33.738565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:33.735761 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:34.187504 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:34.187458 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd"] Oct 30 13:28:34.198745 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:34.198690 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:28:34.198889 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:34.198797 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:28:34.198889 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:34.198846 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:28:34.266785 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:34.266749 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" event={"ID":"415237a2-07c0-481d-bb7b-f818c83e92ce","Type":"ContainerStarted","Data":"df0c1ac3c94e8e70e239fd4bbcaf7f6963798fb310975018de7cfa528dbccc2c"} Oct 30 13:28:35.269707 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.269672 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn"] Oct 30 13:28:35.270534 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.270510 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="51645cac-ccad-44c3-9928-d0283dc273ee" containerName="istio-proxy" Oct 30 13:28:35.274448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.274417 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.276162 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.276133 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" event={"ID":"415237a2-07c0-481d-bb7b-f818c83e92ce","Type":"ContainerStarted","Data":"00061988993a7660d7b7ff0f9c51ee66c0c4a28107ff661c82543a4198ffe954"} Oct 30 13:28:35.291070 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.291038 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-ff4285-openshift-default-dockercfg-4t4rj\"" Oct 30 13:28:35.307086 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.307047 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn"] Oct 30 13:28:35.340029 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.339985 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.340242 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.340178 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.340326 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.340250 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.340393 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.340370 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d669cd-c104-458a-85c2-65d8f7e0c50a-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.340465 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.340426 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.340650 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.340627 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.340720 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.340709 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.340780 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.340740 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khdxv\" (UniqueName: \"kubernetes.io/projected/37d669cd-c104-458a-85c2-65d8f7e0c50a-kube-api-access-khdxv\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.340843 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.340791 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.371242 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.371178 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" podStartSLOduration=2.371163595 podStartE2EDuration="2.371163595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:35.370273868 +0000 UTC m=+2172.002461033" watchObservedRunningTime="2026-10-30 13:28:35.371163595 +0000 UTC m=+2172.003350805" Oct 30 13:28:35.442378 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.442340 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.442670 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.442420 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.442670 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.442445 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khdxv\" (UniqueName: \"kubernetes.io/projected/37d669cd-c104-458a-85c2-65d8f7e0c50a-kube-api-access-khdxv\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.442670 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.442467 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.442670 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.442620 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.442914 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.442739 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.442914 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.442871 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.442991 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.442950 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.443181 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.443153 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.443306 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.443237 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.443382 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.443306 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d669cd-c104-458a-85c2-65d8f7e0c50a-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.443382 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.443338 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.443689 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.443632 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.444350 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.444321 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d669cd-c104-458a-85c2-65d8f7e0c50a-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.445212 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.445181 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.445603 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.445579 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.469980 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.469935 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khdxv\" (UniqueName: \"kubernetes.io/projected/37d669cd-c104-458a-85c2-65d8f7e0c50a-kube-api-access-khdxv\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.470294 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.470270 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.589262 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.589138 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:35.736672 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.736622 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:35.743836 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.743807 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:35.919877 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.919844 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn"] Oct 30 13:28:35.921461 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:28:35.921427 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d669cd_c104_458a_85c2_65d8f7e0c50a.slice/crio-9d7e2bd1c39f707b7d52187ce2af621d48aafb57e11f3441dd6b0c47d008e4aa WatchSource:0}: Error finding container 9d7e2bd1c39f707b7d52187ce2af621d48aafb57e11f3441dd6b0c47d008e4aa: Status 404 returned error can't find the container with id 9d7e2bd1c39f707b7d52187ce2af621d48aafb57e11f3441dd6b0c47d008e4aa Oct 30 13:28:35.924357 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.924322 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:28:35.924454 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.924392 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:28:35.924454 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:35.924422 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:28:36.283694 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:36.283607 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" event={"ID":"37d669cd-c104-458a-85c2-65d8f7e0c50a","Type":"ContainerStarted","Data":"03d17a3ee8584b75c10c98806a8756a141d70b43e174a4df452d6df2f5102ab8"} Oct 30 13:28:36.284154 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:36.283829 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:36.284154 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:36.283877 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" event={"ID":"37d669cd-c104-458a-85c2-65d8f7e0c50a","Type":"ContainerStarted","Data":"9d7e2bd1c39f707b7d52187ce2af621d48aafb57e11f3441dd6b0c47d008e4aa"} Oct 30 13:28:36.284887 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:36.284860 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:28:36.333712 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:36.333645 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" podStartSLOduration=1.333624269 podStartE2EDuration="1.333624269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:36.329247184 +0000 UTC m=+2172.961434350" watchObservedRunningTime="2026-10-30 13:28:36.333624269 +0000 UTC m=+2172.965811434" Oct 30 13:28:36.589830 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:36.589746 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:37.352050 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.351985 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" podUID="3428a596-3e50-418c-9442-9ee08eb312e1" containerName="main" containerID="cri-o://3e8c66f33d2bdb43cf1e5eed13d928ecdc1e2dfb5cd8ed11aab287f26e4b56d4" gracePeriod=45 Oct 30 13:28:37.620559 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.618728 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 13:28:37.620559 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.618802 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" podUID="37d669cd-c104-458a-85c2-65d8f7e0c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 13:28:37.719445 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.719416 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:28:37.778468 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.778424 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-model-cache\") pod \"3428a596-3e50-418c-9442-9ee08eb312e1\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " Oct 30 13:28:37.778680 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.778514 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3428a596-3e50-418c-9442-9ee08eb312e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/3428a596-3e50-418c-9442-9ee08eb312e1-tls-certs\") pod \"3428a596-3e50-418c-9442-9ee08eb312e1\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " Oct 30 13:28:37.778680 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.778560 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-dshm\") pod \"3428a596-3e50-418c-9442-9ee08eb312e1\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " Oct 30 13:28:37.778680 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.778588 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-home\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-home\") pod \"3428a596-3e50-418c-9442-9ee08eb312e1\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " Oct 30 13:28:37.778680 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.778627 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-tmp-dir\") pod \"3428a596-3e50-418c-9442-9ee08eb312e1\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " Oct 30 13:28:37.778907 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.778760 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3428a596-3e50-418c-9442-9ee08eb312e1-kube-api-access-275sx\" (UniqueName: \"kubernetes.io/projected/3428a596-3e50-418c-9442-9ee08eb312e1-kube-api-access-275sx\") pod \"3428a596-3e50-418c-9442-9ee08eb312e1\" (UID: \"3428a596-3e50-418c-9442-9ee08eb312e1\") " Oct 30 13:28:37.780136 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.778512 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-model-cache" pod "3428a596-3e50-418c-9442-9ee08eb312e1" (UID: "3428a596-3e50-418c-9442-9ee08eb312e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:37.780308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.780100 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-home" pod "3428a596-3e50-418c-9442-9ee08eb312e1" (UID: "3428a596-3e50-418c-9442-9ee08eb312e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:37.780685 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.780650 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-tmp-dir" pod "3428a596-3e50-418c-9442-9ee08eb312e1" (UID: "3428a596-3e50-418c-9442-9ee08eb312e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:37.782021 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.781983 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3428a596-3e50-418c-9442-9ee08eb312e1-kube-api-access-275sx" pod "3428a596-3e50-418c-9442-9ee08eb312e1" (UID: "3428a596-3e50-418c-9442-9ee08eb312e1"). InnerVolumeSpecName "kube-api-access-275sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:37.782552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.782524 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-dshm" pod "3428a596-3e50-418c-9442-9ee08eb312e1" (UID: "3428a596-3e50-418c-9442-9ee08eb312e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:37.782660 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.782562 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3428a596-3e50-418c-9442-9ee08eb312e1-tls-certs" pod "3428a596-3e50-418c-9442-9ee08eb312e1" (UID: "3428a596-3e50-418c-9442-9ee08eb312e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:28:37.881265 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.881180 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-275sx\" (UniqueName: \"kubernetes.io/projected/3428a596-3e50-418c-9442-9ee08eb312e1-kube-api-access-275sx\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:37.881265 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.881212 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:37.881265 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.881223 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3428a596-3e50-418c-9442-9ee08eb312e1-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:37.881265 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.881231 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:37.881265 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.881238 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:37.881265 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:37.881246 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3428a596-3e50-418c-9442-9ee08eb312e1-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:38.362571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:38.353188 7010 generic.go:356] "Generic (PLEG): container finished" podID="3428a596-3e50-418c-9442-9ee08eb312e1" containerID="3e8c66f33d2bdb43cf1e5eed13d928ecdc1e2dfb5cd8ed11aab287f26e4b56d4" exitCode=0 Oct 30 13:28:38.362571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:38.353544 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" event={"ID":"3428a596-3e50-418c-9442-9ee08eb312e1","Type":"ContainerDied","Data":"3e8c66f33d2bdb43cf1e5eed13d928ecdc1e2dfb5cd8ed11aab287f26e4b56d4"} Oct 30 13:28:38.362571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:38.353602 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" event={"ID":"3428a596-3e50-418c-9442-9ee08eb312e1","Type":"ContainerDied","Data":"7d658808dbcde7eaa5ff1bdcbb73d049a61372c564029eea2a23b7b1a3041c45"} Oct 30 13:28:38.362571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:38.353626 7010 scope.go:122] "RemoveContainer" containerID="3e8c66f33d2bdb43cf1e5eed13d928ecdc1e2dfb5cd8ed11aab287f26e4b56d4" Oct 30 13:28:38.362571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:38.353790 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz" Oct 30 13:28:38.470560 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:38.466640 7010 scope.go:122] "RemoveContainer" containerID="3e8c66f33d2bdb43cf1e5eed13d928ecdc1e2dfb5cd8ed11aab287f26e4b56d4" Oct 30 13:28:38.475613 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:28:38.474642 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e8c66f33d2bdb43cf1e5eed13d928ecdc1e2dfb5cd8ed11aab287f26e4b56d4\": container with ID starting with 3e8c66f33d2bdb43cf1e5eed13d928ecdc1e2dfb5cd8ed11aab287f26e4b56d4 not found: ID does not exist" containerID="3e8c66f33d2bdb43cf1e5eed13d928ecdc1e2dfb5cd8ed11aab287f26e4b56d4" Oct 30 13:28:38.475613 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:38.474690 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e8c66f33d2bdb43cf1e5eed13d928ecdc1e2dfb5cd8ed11aab287f26e4b56d4"} err="failed to get container status \"3e8c66f33d2bdb43cf1e5eed13d928ecdc1e2dfb5cd8ed11aab287f26e4b56d4\": rpc error: code = NotFound desc = could not find container \"3e8c66f33d2bdb43cf1e5eed13d928ecdc1e2dfb5cd8ed11aab287f26e4b56d4\": container with ID starting with 3e8c66f33d2bdb43cf1e5eed13d928ecdc1e2dfb5cd8ed11aab287f26e4b56d4 not found: ID does not exist" Oct 30 13:28:38.535021 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:38.534600 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz"] Oct 30 13:28:38.540009 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:38.538755 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5c7ba3/test-model-5c7ba3-kserve-5c576d9f49-hr4wz"] Oct 30 13:28:38.598515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:38.590924 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 13:28:38.598515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:38.590982 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" podUID="37d669cd-c104-458a-85c2-65d8f7e0c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 13:28:39.524845 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:39.524805 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3428a596-3e50-418c-9442-9ee08eb312e1" path="/var/lib/kubelet/pods/3428a596-3e50-418c-9442-9ee08eb312e1/volumes" Oct 30 13:28:39.597513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:39.590939 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 13:28:39.597513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:39.591004 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" podUID="37d669cd-c104-458a-85c2-65d8f7e0c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 13:28:40.603514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:40.603142 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 13:28:40.603514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:40.603203 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" podUID="37d669cd-c104-458a-85c2-65d8f7e0c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 13:28:41.611569 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:41.605641 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 13:28:41.611569 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:41.605719 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" podUID="37d669cd-c104-458a-85c2-65d8f7e0c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 13:28:42.540513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:42.539703 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw"] Oct 30 13:28:42.540737 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:42.540364 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" podUID="84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" containerName="istio-proxy" containerID="cri-o://5f1f0d55567654d8d74c71e7481622d713fdf03663d29f8d28f21c2bb1a47b22" gracePeriod=30 Oct 30 13:28:42.592418 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:42.591256 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 13:28:42.592418 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:42.591336 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" podUID="37d669cd-c104-458a-85c2-65d8f7e0c50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 13:28:43.595696 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:43.595660 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:43.596188 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:43.596065 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:43.597072 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:43.597050 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:28:46.133207 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:46.133173 7010 patch_prober.go:29] interesting pod/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:28:46.133750 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:46.133227 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" podUID="84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:28:47.656609 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:28:47.656482 7010 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b36af0_e09d_4810_b85c_e5ed5e5e2ee6.slice/crio-5f1f0d55567654d8d74c71e7481622d713fdf03663d29f8d28f21c2bb1a47b22.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b36af0_e09d_4810_b85c_e5ed5e5e2ee6.slice/crio-conmon-5f1f0d55567654d8d74c71e7481622d713fdf03663d29f8d28f21c2bb1a47b22.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:28:47.849273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.849238 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:28:47.953538 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.953445 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-data\") pod \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " Oct 30 13:28:47.953737 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.953602 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-envoy\") pod \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " Oct 30 13:28:47.953737 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.953634 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-podinfo\") pod \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " Oct 30 13:28:47.953737 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.953689 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-token\" (UniqueName: \"kubernetes.io/projected/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-token\") pod \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " Oct 30 13:28:47.953912 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.953732 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-data" pod "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" (UID: "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:47.953912 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.953741 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-workload-certs\") pod \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " Oct 30 13:28:47.953912 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.953884 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-workload-socket\") pod \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " Oct 30 13:28:47.954066 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.953913 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istiod-ca-cert\") pod \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " Oct 30 13:28:47.954066 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.954021 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-workload-certs" pod "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" (UID: "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:47.954168 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.954069 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-credential-socket\") pod \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " Oct 30 13:28:47.954168 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.954115 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-kube-api-access-n95pl\" (UniqueName: \"kubernetes.io/projected/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-kube-api-access-n95pl\") pod \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\" (UID: \"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6\") " Oct 30 13:28:47.954168 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.954123 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-workload-socket" pod "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" (UID: "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:47.954329 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.954310 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-credential-socket" pod "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" (UID: "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:47.954390 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.954350 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istiod-ca-cert" pod "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" (UID: "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:28:47.954817 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.954790 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:47.954921 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.954820 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:47.954921 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.954836 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:47.954921 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.954850 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:47.954921 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.954866 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:47.956008 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.955983 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-podinfo" pod "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" (UID: "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:28:47.956106 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.956004 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-envoy" pod "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" (UID: "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:28:47.956164 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.956140 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-token" pod "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" (UID: "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:47.956286 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:47.956263 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-kube-api-access-n95pl" pod "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" (UID: "84b36af0-e09d-4810-b85c-e5ed5e5e2ee6"). InnerVolumeSpecName "kube-api-access-n95pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:28:48.056286 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:48.056243 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n95pl\" (UniqueName: \"kubernetes.io/projected/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-kube-api-access-n95pl\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:48.056286 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:48.056279 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:48.056286 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:48.056290 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:48.056286 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:48.056300 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:28:48.540679 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:48.540644 7010 generic.go:356] "Generic (PLEG): container finished" podID="84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" containerID="5f1f0d55567654d8d74c71e7481622d713fdf03663d29f8d28f21c2bb1a47b22" exitCode=0 Oct 30 13:28:48.540877 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:48.540693 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" Oct 30 13:28:48.540877 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:48.540724 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" event={"ID":"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6","Type":"ContainerDied","Data":"5f1f0d55567654d8d74c71e7481622d713fdf03663d29f8d28f21c2bb1a47b22"} Oct 30 13:28:48.540877 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:48.540781 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw" event={"ID":"84b36af0-e09d-4810-b85c-e5ed5e5e2ee6","Type":"ContainerDied","Data":"76cfaf9f99f3c1cc25b329782881c6a07b3ab4008ac536b87c1969c6650b396d"} Oct 30 13:28:48.540877 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:48.540805 7010 scope.go:122] "RemoveContainer" containerID="5f1f0d55567654d8d74c71e7481622d713fdf03663d29f8d28f21c2bb1a47b22" Oct 30 13:28:48.552314 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:48.552294 7010 scope.go:122] "RemoveContainer" containerID="5f1f0d55567654d8d74c71e7481622d713fdf03663d29f8d28f21c2bb1a47b22" Oct 30 13:28:48.552734 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:28:48.552711 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1f0d55567654d8d74c71e7481622d713fdf03663d29f8d28f21c2bb1a47b22\": container with ID starting with 5f1f0d55567654d8d74c71e7481622d713fdf03663d29f8d28f21c2bb1a47b22 not found: ID does not exist" containerID="5f1f0d55567654d8d74c71e7481622d713fdf03663d29f8d28f21c2bb1a47b22" Oct 30 13:28:48.552833 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:48.552744 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1f0d55567654d8d74c71e7481622d713fdf03663d29f8d28f21c2bb1a47b22"} err="failed to get container status \"5f1f0d55567654d8d74c71e7481622d713fdf03663d29f8d28f21c2bb1a47b22\": rpc error: code = NotFound desc = could not find container \"5f1f0d55567654d8d74c71e7481622d713fdf03663d29f8d28f21c2bb1a47b22\": container with ID starting with 5f1f0d55567654d8d74c71e7481622d713fdf03663d29f8d28f21c2bb1a47b22 not found: ID does not exist" Oct 30 13:28:48.609895 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:48.609859 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw"] Oct 30 13:28:48.623285 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:48.623257 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e34fd5-openshift-default-59fc7bd49-rh2rw"] Oct 30 13:28:49.489799 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:28:49.489768 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" path="/var/lib/kubelet/pods/84b36af0-e09d-4810-b85c-e5ed5e5e2ee6/volumes" Oct 30 13:28:59.403662 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:28:59.403617 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:29:02.259599 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.259562 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz"] Oct 30 13:29:02.260407 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.260383 7010 cpu_manager.go:414] "containerMap: removing container" podUID="84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" containerName="istio-proxy" Oct 30 13:29:02.260407 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.260406 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" containerName="istio-proxy" Oct 30 13:29:02.260619 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.260441 7010 cpu_manager.go:414] "containerMap: removing container" podUID="3428a596-3e50-418c-9442-9ee08eb312e1" containerName="main" Oct 30 13:29:02.260619 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.260450 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3428a596-3e50-418c-9442-9ee08eb312e1" containerName="main" Oct 30 13:29:02.260723 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.260678 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="3428a596-3e50-418c-9442-9ee08eb312e1" containerName="main" Oct 30 13:29:02.260723 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.260709 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b36af0-e09d-4810-b85c-e5ed5e5e2ee6" containerName="istio-proxy" Oct 30 13:29:02.266004 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.265982 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.299309 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.299269 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz"] Oct 30 13:29:02.355824 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.355786 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.356002 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.355833 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.356002 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.355922 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l7g2\" (UniqueName: \"kubernetes.io/projected/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-kube-api-access-7l7g2\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.356118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.356097 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.356206 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.356187 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.356269 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.356250 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.356314 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.356302 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.356396 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.356383 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.356432 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.356415 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.457850 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.457809 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.458028 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.457880 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.458028 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.457918 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.458028 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.457953 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.458171 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.458032 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.458171 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.458071 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.458171 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.458165 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.458333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.458210 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.458333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.458259 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l7g2\" (UniqueName: \"kubernetes.io/projected/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-kube-api-access-7l7g2\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.458333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.458317 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.458602 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.458577 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.458743 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.458714 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.458802 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.458751 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.458802 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.458717 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.460310 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.460287 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.460719 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.460696 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.482412 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.482380 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l7g2\" (UniqueName: \"kubernetes.io/projected/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-kube-api-access-7l7g2\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.482592 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.482385 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.581008 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.580925 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:02.913823 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.913789 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz"] Oct 30 13:29:02.916270 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:29:02.916232 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ffbc852_329e_4a5a_a9ea_0cba3a63ec6e.slice/crio-0a4dd2b24ec73684a9742f3121ca4a22ae0e3401b88dfd1c7a1564f0a3f0deda WatchSource:0}: Error finding container 0a4dd2b24ec73684a9742f3121ca4a22ae0e3401b88dfd1c7a1564f0a3f0deda: Status 404 returned error can't find the container with id 0a4dd2b24ec73684a9742f3121ca4a22ae0e3401b88dfd1c7a1564f0a3f0deda Oct 30 13:29:02.919848 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.919812 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:29:02.919966 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.919902 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:29:02.919966 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:02.919929 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:29:03.661805 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:03.661757 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" event={"ID":"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e","Type":"ContainerStarted","Data":"a2f8af7ed34db1bc7f9d281b72d87aeeaf4efe29d638fe85b6b1b770efa30790"} Oct 30 13:29:03.662330 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:03.661818 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" event={"ID":"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e","Type":"ContainerStarted","Data":"0a4dd2b24ec73684a9742f3121ca4a22ae0e3401b88dfd1c7a1564f0a3f0deda"} Oct 30 13:29:03.742421 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:03.742350 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" podStartSLOduration=1.7423324519999999 podStartE2EDuration="1.742332452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:03.727936686 +0000 UTC m=+2200.360123847" watchObservedRunningTime="2026-10-30 13:29:03.742332452 +0000 UTC m=+2200.374519614" Oct 30 13:29:04.055233 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.055155 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm"] Oct 30 13:29:04.059472 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.059446 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.088867 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.088828 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-376c89-openshift-default-dockercfg-twmvq\"" Oct 30 13:29:04.112768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.112724 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm"] Oct 30 13:29:04.117689 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.117580 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e516fd-dda7-4d15-8562-6050961291aa-istio-token\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.117885 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.117708 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e516fd-dda7-4d15-8562-6050961291aa-istio-podinfo\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.117885 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.117870 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e516fd-dda7-4d15-8562-6050961291aa-istiod-ca-cert\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.118012 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.117911 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-workload-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.118012 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.117954 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-istio-envoy\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.118121 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.118015 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-istio-data\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.118121 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.118105 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-credential-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.118238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.118137 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9w8z\" (UniqueName: \"kubernetes.io/projected/19e516fd-dda7-4d15-8562-6050961291aa-kube-api-access-g9w8z\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.118238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.118161 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-workload-certs\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.219695 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.219654 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e516fd-dda7-4d15-8562-6050961291aa-istiod-ca-cert\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.219887 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.219713 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-workload-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.219887 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.219747 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-istio-envoy\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.219887 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.219794 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-istio-data\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.219887 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.219862 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-credential-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.220109 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.219890 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9w8z\" (UniqueName: \"kubernetes.io/projected/19e516fd-dda7-4d15-8562-6050961291aa-kube-api-access-g9w8z\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.220109 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.219918 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-workload-certs\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.220109 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.220038 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e516fd-dda7-4d15-8562-6050961291aa-istio-token\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.220109 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.220087 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e516fd-dda7-4d15-8562-6050961291aa-istio-podinfo\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.220301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.220260 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-credential-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.220301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.220269 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-istio-data\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.220389 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.220378 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-workload-socket\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.220575 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.220547 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-workload-certs\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.221027 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.221002 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e516fd-dda7-4d15-8562-6050961291aa-istiod-ca-cert\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.223750 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.223729 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-istio-envoy\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.224055 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.224036 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e516fd-dda7-4d15-8562-6050961291aa-istio-podinfo\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.246603 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.246571 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9w8z\" (UniqueName: \"kubernetes.io/projected/19e516fd-dda7-4d15-8562-6050961291aa-kube-api-access-g9w8z\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.246794 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.246671 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e516fd-dda7-4d15-8562-6050961291aa-istio-token\") pod \"e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.374477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.374444 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:04.581898 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.581863 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:04.583200 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.583172 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 13:29:04.583346 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.583228 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" podUID="7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 13:29:04.742071 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.742043 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm"] Oct 30 13:29:04.743785 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:29:04.743758 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e516fd_dda7_4d15_8562_6050961291aa.slice/crio-7c9e15e267b19b9cb311ea207325fdd78c0a4172d1a38876cd013393f030a9af WatchSource:0}: Error finding container 7c9e15e267b19b9cb311ea207325fdd78c0a4172d1a38876cd013393f030a9af: Status 404 returned error can't find the container with id 7c9e15e267b19b9cb311ea207325fdd78c0a4172d1a38876cd013393f030a9af Oct 30 13:29:04.746867 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.746830 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:29:04.746974 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.746931 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:29:04.746974 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:04.746970 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:29:05.581869 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:05.581836 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 13:29:05.582100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:05.581889 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" podUID="7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 13:29:05.675450 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:05.675410 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" event={"ID":"19e516fd-dda7-4d15-8562-6050961291aa","Type":"ContainerStarted","Data":"b4840f0accf48d6bb59b48b4d46fab74e7671b5f376feb3994c7ce27b7e988ab"} Oct 30 13:29:05.675660 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:05.675471 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" event={"ID":"19e516fd-dda7-4d15-8562-6050961291aa","Type":"ContainerStarted","Data":"7c9e15e267b19b9cb311ea207325fdd78c0a4172d1a38876cd013393f030a9af"} Oct 30 13:29:06.375565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:06.375526 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:06.376737 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:06.376710 7010 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 13:29:06.376933 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:06.376906 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" podUID="19e516fd-dda7-4d15-8562-6050961291aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 13:29:06.581542 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:06.581460 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 13:29:06.581731 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:06.581562 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" podUID="7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 13:29:07.375388 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:07.375348 7010 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 13:29:07.375615 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:07.375400 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" podUID="19e516fd-dda7-4d15-8562-6050961291aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 13:29:07.581867 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:07.581834 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 13:29:07.582087 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:07.582058 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" podUID="7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 13:29:08.375481 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:08.375442 7010 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 13:29:08.375817 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:08.375786 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" podUID="19e516fd-dda7-4d15-8562-6050961291aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 13:29:08.581283 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:08.581244 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 13:29:08.582709 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:08.582672 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" podUID="7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 13:29:09.374978 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:09.374943 7010 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 13:29:09.375173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:09.374991 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" podUID="19e516fd-dda7-4d15-8562-6050961291aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 13:29:09.581929 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:09.581885 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 13:29:09.582340 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:09.581977 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" podUID="7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 13:29:10.374905 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:10.374868 7010 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 13:29:10.375127 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:10.374944 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" podUID="19e516fd-dda7-4d15-8562-6050961291aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 13:29:10.679917 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:10.679828 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:10.681108 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:10.681082 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:10.681680 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:10.681655 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:29:10.747143 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:10.747068 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" podStartSLOduration=6.747048528 podStartE2EDuration="6.747048528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:05.769007115 +0000 UTC m=+2202.401194278" watchObservedRunningTime="2026-10-30 13:29:10.747048528 +0000 UTC m=+2207.379235692" Oct 30 13:29:11.375758 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:11.375707 7010 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 13:29:11.375963 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:11.375781 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" podUID="19e516fd-dda7-4d15-8562-6050961291aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 13:29:11.750381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:11.750286 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66"] Oct 30 13:29:11.750870 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:11.750527 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" podUID="03a68869-2b11-4796-a767-ab7402efbf87" containerName="istio-proxy" containerID="cri-o://46cc19fec274a9ef0339e3a5bbeb807b27f8599db8d87e9b0967756b0a53c20d" gracePeriod=30 Oct 30 13:29:12.151817 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.151780 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst"] Oct 30 13:29:12.152045 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.152024 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" podUID="be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a" containerName="maas-api" containerID="cri-o://4ab0dba59438fc0533b82450fe813d54d87247111e6e2afe0ccdcf3ca7eef4bb" gracePeriod=30 Oct 30 13:29:12.228687 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.228626 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" podUID="be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.104:8443/health\": dial tcp 10.134.0.104:8443: connect: connection refused" Oct 30 13:29:12.403933 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.402279 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:12.407431 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.406859 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:12.409285 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.409258 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:29:12.549395 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.548962 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:29:12.577987 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.577950 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-maas-api-tls\") pod \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\" (UID: \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\") " Oct 30 13:29:12.578173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.578013 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-metrics-certs\") pod \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\" (UID: \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\") " Oct 30 13:29:12.578173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.578061 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-openshift-service-ca\") pod \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\" (UID: \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\") " Oct 30 13:29:12.578339 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.578322 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-kube-api-access-mncvk\" (UniqueName: \"kubernetes.io/projected/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-kube-api-access-mncvk\") pod \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\" (UID: \"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a\") " Oct 30 13:29:12.585723 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.585635 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-openshift-service-ca" pod "be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a" (UID: "be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:12.592622 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.587832 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-metrics-certs" pod "be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a" (UID: "be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:29:12.592906 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.592808 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-maas-api-tls" pod "be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a" (UID: "be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:29:12.602967 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.602872 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-kube-api-access-mncvk" pod "be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a" (UID: "be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a"). InnerVolumeSpecName "kube-api-access-mncvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:12.686129 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.686033 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mncvk\" (UniqueName: \"kubernetes.io/projected/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-kube-api-access-mncvk\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:12.686129 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.686070 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-maas-api-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:12.686129 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.686080 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:12.686129 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.686089 7010 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a-openshift-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:12.748734 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.748681 7010 generic.go:356] "Generic (PLEG): container finished" podID="be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a" containerID="4ab0dba59438fc0533b82450fe813d54d87247111e6e2afe0ccdcf3ca7eef4bb" exitCode=0 Oct 30 13:29:12.748950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.748784 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" event={"ID":"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a","Type":"ContainerDied","Data":"4ab0dba59438fc0533b82450fe813d54d87247111e6e2afe0ccdcf3ca7eef4bb"} Oct 30 13:29:12.748950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.748867 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" event={"ID":"be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a","Type":"ContainerDied","Data":"06854218ad3559af5f3bbd0dadb03d9be7621c5818701bf71d937409c7035bfb"} Oct 30 13:29:12.748950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.748914 7010 scope.go:122] "RemoveContainer" containerID="4ab0dba59438fc0533b82450fe813d54d87247111e6e2afe0ccdcf3ca7eef4bb" Oct 30 13:29:12.754382 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.749121 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst" Oct 30 13:29:12.786753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.783737 7010 scope.go:122] "RemoveContainer" containerID="4ab0dba59438fc0533b82450fe813d54d87247111e6e2afe0ccdcf3ca7eef4bb" Oct 30 13:29:12.786753 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:29:12.784155 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab0dba59438fc0533b82450fe813d54d87247111e6e2afe0ccdcf3ca7eef4bb\": container with ID starting with 4ab0dba59438fc0533b82450fe813d54d87247111e6e2afe0ccdcf3ca7eef4bb not found: ID does not exist" containerID="4ab0dba59438fc0533b82450fe813d54d87247111e6e2afe0ccdcf3ca7eef4bb" Oct 30 13:29:12.786753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.784259 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab0dba59438fc0533b82450fe813d54d87247111e6e2afe0ccdcf3ca7eef4bb"} err="failed to get container status \"4ab0dba59438fc0533b82450fe813d54d87247111e6e2afe0ccdcf3ca7eef4bb\": rpc error: code = NotFound desc = could not find container \"4ab0dba59438fc0533b82450fe813d54d87247111e6e2afe0ccdcf3ca7eef4bb\": container with ID starting with 4ab0dba59438fc0533b82450fe813d54d87247111e6e2afe0ccdcf3ca7eef4bb not found: ID does not exist" Oct 30 13:29:12.844124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.844087 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst"] Oct 30 13:29:12.854840 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:12.854800 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4651e8-b6896f56d-xdcst"] Oct 30 13:29:13.271365 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:13.271325 7010 patch_prober.go:29] interesting pod/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:29:13.271736 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:13.271692 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" podUID="03a68869-2b11-4796-a767-ab7402efbf87" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:29:13.500034 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:13.499989 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a" path="/var/lib/kubelet/pods/be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a/volumes" Oct 30 13:29:17.062003 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.061972 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:29:17.151157 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.151112 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a68869-2b11-4796-a767-ab7402efbf87-istio-token\" (UniqueName: \"kubernetes.io/projected/03a68869-2b11-4796-a767-ab7402efbf87-istio-token\") pod \"03a68869-2b11-4796-a767-ab7402efbf87\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " Oct 30 13:29:17.151397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.151199 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03a68869-2b11-4796-a767-ab7402efbf87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a68869-2b11-4796-a767-ab7402efbf87-istio-podinfo\") pod \"03a68869-2b11-4796-a767-ab7402efbf87\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " Oct 30 13:29:17.151397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.151290 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-workload-certs\") pod \"03a68869-2b11-4796-a767-ab7402efbf87\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " Oct 30 13:29:17.151397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.151325 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a68869-2b11-4796-a767-ab7402efbf87-kube-api-access-6hxvk\" (UniqueName: \"kubernetes.io/projected/03a68869-2b11-4796-a767-ab7402efbf87-kube-api-access-6hxvk\") pod \"03a68869-2b11-4796-a767-ab7402efbf87\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " Oct 30 13:29:17.151397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.151371 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-istio-data\") pod \"03a68869-2b11-4796-a767-ab7402efbf87\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " Oct 30 13:29:17.151397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.151398 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-credential-socket\") pod \"03a68869-2b11-4796-a767-ab7402efbf87\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " Oct 30 13:29:17.151720 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.151434 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-istio-envoy\") pod \"03a68869-2b11-4796-a767-ab7402efbf87\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " Oct 30 13:29:17.151720 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.151474 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-workload-socket\") pod \"03a68869-2b11-4796-a767-ab7402efbf87\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " Oct 30 13:29:17.151720 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.151516 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a68869-2b11-4796-a767-ab7402efbf87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a68869-2b11-4796-a767-ab7402efbf87-istiod-ca-cert\") pod \"03a68869-2b11-4796-a767-ab7402efbf87\" (UID: \"03a68869-2b11-4796-a767-ab7402efbf87\") " Oct 30 13:29:17.152264 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.152197 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-credential-socket" pod "03a68869-2b11-4796-a767-ab7402efbf87" (UID: "03a68869-2b11-4796-a767-ab7402efbf87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:17.152565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.152403 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a68869-2b11-4796-a767-ab7402efbf87-istiod-ca-cert" pod "03a68869-2b11-4796-a767-ab7402efbf87" (UID: "03a68869-2b11-4796-a767-ab7402efbf87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:17.152873 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.152688 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-istio-data" pod "03a68869-2b11-4796-a767-ab7402efbf87" (UID: "03a68869-2b11-4796-a767-ab7402efbf87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:17.152972 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.152923 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-workload-certs" pod "03a68869-2b11-4796-a767-ab7402efbf87" (UID: "03a68869-2b11-4796-a767-ab7402efbf87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:17.153187 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.153151 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-workload-socket" pod "03a68869-2b11-4796-a767-ab7402efbf87" (UID: "03a68869-2b11-4796-a767-ab7402efbf87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:17.155475 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.155434 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03a68869-2b11-4796-a767-ab7402efbf87-istio-podinfo" pod "03a68869-2b11-4796-a767-ab7402efbf87" (UID: "03a68869-2b11-4796-a767-ab7402efbf87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:29:17.155991 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.155962 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a68869-2b11-4796-a767-ab7402efbf87-kube-api-access-6hxvk" pod "03a68869-2b11-4796-a767-ab7402efbf87" (UID: "03a68869-2b11-4796-a767-ab7402efbf87"). InnerVolumeSpecName "kube-api-access-6hxvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:17.156105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.156046 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-istio-envoy" pod "03a68869-2b11-4796-a767-ab7402efbf87" (UID: "03a68869-2b11-4796-a767-ab7402efbf87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:29:17.157471 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.157444 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a68869-2b11-4796-a767-ab7402efbf87-istio-token" pod "03a68869-2b11-4796-a767-ab7402efbf87" (UID: "03a68869-2b11-4796-a767-ab7402efbf87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:17.253714 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.253676 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a68869-2b11-4796-a767-ab7402efbf87-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.253714 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.253715 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a68869-2b11-4796-a767-ab7402efbf87-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.253965 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.253730 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.253965 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.253746 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hxvk\" (UniqueName: \"kubernetes.io/projected/03a68869-2b11-4796-a767-ab7402efbf87-kube-api-access-6hxvk\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.253965 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.253759 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.253965 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.253767 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.253965 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.253775 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.253965 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.253783 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a68869-2b11-4796-a767-ab7402efbf87-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.253965 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.253791 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a68869-2b11-4796-a767-ab7402efbf87-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:17.783000 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.782892 7010 generic.go:356] "Generic (PLEG): container finished" podID="03a68869-2b11-4796-a767-ab7402efbf87" containerID="46cc19fec274a9ef0339e3a5bbeb807b27f8599db8d87e9b0967756b0a53c20d" exitCode=0 Oct 30 13:29:17.783000 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.782952 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" Oct 30 13:29:17.783000 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.782981 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" event={"ID":"03a68869-2b11-4796-a767-ab7402efbf87","Type":"ContainerDied","Data":"46cc19fec274a9ef0339e3a5bbeb807b27f8599db8d87e9b0967756b0a53c20d"} Oct 30 13:29:17.783285 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.783059 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66" event={"ID":"03a68869-2b11-4796-a767-ab7402efbf87","Type":"ContainerDied","Data":"31d9108b250eaac4282d3102d881557bafe4a7258c5ee9cbcff1536c56e600e1"} Oct 30 13:29:17.783285 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.783082 7010 scope.go:122] "RemoveContainer" containerID="46cc19fec274a9ef0339e3a5bbeb807b27f8599db8d87e9b0967756b0a53c20d" Oct 30 13:29:17.793858 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.793830 7010 scope.go:122] "RemoveContainer" containerID="46cc19fec274a9ef0339e3a5bbeb807b27f8599db8d87e9b0967756b0a53c20d" Oct 30 13:29:17.794145 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:29:17.794127 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46cc19fec274a9ef0339e3a5bbeb807b27f8599db8d87e9b0967756b0a53c20d\": container with ID starting with 46cc19fec274a9ef0339e3a5bbeb807b27f8599db8d87e9b0967756b0a53c20d not found: ID does not exist" containerID="46cc19fec274a9ef0339e3a5bbeb807b27f8599db8d87e9b0967756b0a53c20d" Oct 30 13:29:17.794205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.794154 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46cc19fec274a9ef0339e3a5bbeb807b27f8599db8d87e9b0967756b0a53c20d"} err="failed to get container status \"46cc19fec274a9ef0339e3a5bbeb807b27f8599db8d87e9b0967756b0a53c20d\": rpc error: code = NotFound desc = could not find container \"46cc19fec274a9ef0339e3a5bbeb807b27f8599db8d87e9b0967756b0a53c20d\": container with ID starting with 46cc19fec274a9ef0339e3a5bbeb807b27f8599db8d87e9b0967756b0a53c20d not found: ID does not exist" Oct 30 13:29:17.843399 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.843356 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66"] Oct 30 13:29:17.855649 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:17.855616 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-47628d-openshift-default-65ddb6bd77-dfv66"] Oct 30 13:29:19.489501 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:19.489444 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a68869-2b11-4796-a767-ab7402efbf87" path="/var/lib/kubelet/pods/03a68869-2b11-4796-a767-ab7402efbf87/volumes" Oct 30 13:29:22.599462 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:22.599418 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl"] Oct 30 13:29:22.630331 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:22.630290 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk"] Oct 30 13:29:27.599801 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:27.599755 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" podUID="353779fe-c229-48e5-b17b-095bd6c32f59" containerName="payload-processing" containerID="cri-o://ef437412f0788ce10ba1f38302921da48c4a016f1a808a3bd0925c9b665d2636" gracePeriod=25 Oct 30 13:29:27.631455 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:27.631408 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" podUID="7459ef64-d4fa-4130-acd8-4a0162654b1d" containerName="payload-pre-processing" containerID="cri-o://b7ccb5d3b14ad984fc68fcdb5cf57281abec7877e79bc560ac546d601c38630e" gracePeriod=25 Oct 30 13:29:27.868534 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:27.868459 7010 generic.go:356] "Generic (PLEG): container finished" podID="353779fe-c229-48e5-b17b-095bd6c32f59" containerID="ef437412f0788ce10ba1f38302921da48c4a016f1a808a3bd0925c9b665d2636" exitCode=0 Oct 30 13:29:27.868688 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:27.868665 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" event={"ID":"353779fe-c229-48e5-b17b-095bd6c32f59","Type":"ContainerDied","Data":"ef437412f0788ce10ba1f38302921da48c4a016f1a808a3bd0925c9b665d2636"} Oct 30 13:29:27.868738 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:27.868720 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" event={"ID":"353779fe-c229-48e5-b17b-095bd6c32f59","Type":"ContainerDied","Data":"7cd89c0fee4fcb432c4cd2057f98ed0539f9b2065013de092ebbace104bcab3e"} Oct 30 13:29:27.868786 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:27.868741 7010 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cd89c0fee4fcb432c4cd2057f98ed0539f9b2065013de092ebbace104bcab3e" Oct 30 13:29:27.889781 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:27.889746 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" Oct 30 13:29:27.913881 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:27.913842 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353779fe-c229-48e5-b17b-095bd6c32f59-kube-api-access-4w9cc\" (UniqueName: \"kubernetes.io/projected/353779fe-c229-48e5-b17b-095bd6c32f59-kube-api-access-4w9cc\") pod \"353779fe-c229-48e5-b17b-095bd6c32f59\" (UID: \"353779fe-c229-48e5-b17b-095bd6c32f59\") " Oct 30 13:29:27.914066 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:27.913963 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/353779fe-c229-48e5-b17b-095bd6c32f59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353779fe-c229-48e5-b17b-095bd6c32f59-plugins-config-volume\") pod \"353779fe-c229-48e5-b17b-095bd6c32f59\" (UID: \"353779fe-c229-48e5-b17b-095bd6c32f59\") " Oct 30 13:29:27.914318 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:27.914296 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353779fe-c229-48e5-b17b-095bd6c32f59-plugins-config-volume" pod "353779fe-c229-48e5-b17b-095bd6c32f59" (UID: "353779fe-c229-48e5-b17b-095bd6c32f59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:27.916041 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:27.916022 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353779fe-c229-48e5-b17b-095bd6c32f59-kube-api-access-4w9cc" pod "353779fe-c229-48e5-b17b-095bd6c32f59" (UID: "353779fe-c229-48e5-b17b-095bd6c32f59"). InnerVolumeSpecName "kube-api-access-4w9cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:28.015665 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:28.015629 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w9cc\" (UniqueName: \"kubernetes.io/projected/353779fe-c229-48e5-b17b-095bd6c32f59-kube-api-access-4w9cc\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:28.015665 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:28.015657 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353779fe-c229-48e5-b17b-095bd6c32f59-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:28.876301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:28.876263 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl" Oct 30 13:29:28.931244 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:28.931206 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl"] Oct 30 13:29:28.943707 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:28.943675 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5c7ba3-566f6d6fc6-kzzxl"] Oct 30 13:29:29.404589 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:29:29.404552 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:29:29.489545 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:29.489507 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="353779fe-c229-48e5-b17b-095bd6c32f59" path="/var/lib/kubelet/pods/353779fe-c229-48e5-b17b-095bd6c32f59/volumes" Oct 30 13:29:32.049576 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:32.049535 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm"] Oct 30 13:29:32.080777 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:32.080738 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws"] Oct 30 13:29:34.686646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:34.686604 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" podUID="7459ef64-d4fa-4130-acd8-4a0162654b1d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:29:37.050107 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:37.050054 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" podUID="d1c819c5-dff1-4b26-a66f-d4d2f6376387" containerName="payload-processing" containerID="cri-o://6d7e784beb25ef5803d3587e7e3992460f6357cbaae9f0085f8129152a489a1a" gracePeriod=25 Oct 30 13:29:37.081562 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:37.081484 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" podUID="ef58e88a-0bad-4099-9194-d4cd16997478" containerName="payload-pre-processing" containerID="cri-o://460db5f17ee38a8decce8c834e64196aac68f83abb4881848d4ab63c4361dba4" gracePeriod=25 Oct 30 13:29:37.379291 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:37.379261 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" Oct 30 13:29:37.468904 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:37.468848 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c819c5-dff1-4b26-a66f-d4d2f6376387-kube-api-access-z46ql\" (UniqueName: \"kubernetes.io/projected/d1c819c5-dff1-4b26-a66f-d4d2f6376387-kube-api-access-z46ql\") pod \"d1c819c5-dff1-4b26-a66f-d4d2f6376387\" (UID: \"d1c819c5-dff1-4b26-a66f-d4d2f6376387\") " Oct 30 13:29:37.469110 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:37.468918 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c819c5-dff1-4b26-a66f-d4d2f6376387-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c819c5-dff1-4b26-a66f-d4d2f6376387-plugins-config-volume\") pod \"d1c819c5-dff1-4b26-a66f-d4d2f6376387\" (UID: \"d1c819c5-dff1-4b26-a66f-d4d2f6376387\") " Oct 30 13:29:37.469454 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:37.469420 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c819c5-dff1-4b26-a66f-d4d2f6376387-plugins-config-volume" pod "d1c819c5-dff1-4b26-a66f-d4d2f6376387" (UID: "d1c819c5-dff1-4b26-a66f-d4d2f6376387"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:37.469653 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:37.469631 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c819c5-dff1-4b26-a66f-d4d2f6376387-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:37.471219 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:37.471188 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c819c5-dff1-4b26-a66f-d4d2f6376387-kube-api-access-z46ql" pod "d1c819c5-dff1-4b26-a66f-d4d2f6376387" (UID: "d1c819c5-dff1-4b26-a66f-d4d2f6376387"). InnerVolumeSpecName "kube-api-access-z46ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:37.570608 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:37.570569 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z46ql\" (UniqueName: \"kubernetes.io/projected/d1c819c5-dff1-4b26-a66f-d4d2f6376387-kube-api-access-z46ql\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:37.935275 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:37.935233 7010 generic.go:356] "Generic (PLEG): container finished" podID="d1c819c5-dff1-4b26-a66f-d4d2f6376387" containerID="6d7e784beb25ef5803d3587e7e3992460f6357cbaae9f0085f8129152a489a1a" exitCode=0 Oct 30 13:29:37.935450 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:37.935299 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" Oct 30 13:29:37.935450 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:37.935304 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" event={"ID":"d1c819c5-dff1-4b26-a66f-d4d2f6376387","Type":"ContainerDied","Data":"6d7e784beb25ef5803d3587e7e3992460f6357cbaae9f0085f8129152a489a1a"} Oct 30 13:29:37.935450 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:37.935392 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm" event={"ID":"d1c819c5-dff1-4b26-a66f-d4d2f6376387","Type":"ContainerDied","Data":"6e296f3146a62a7c821ada4556cc249027ec825dc6e0f55bff890fe9f9bd2ad0"} Oct 30 13:29:37.935450 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:37.935418 7010 scope.go:122] "RemoveContainer" containerID="6d7e784beb25ef5803d3587e7e3992460f6357cbaae9f0085f8129152a489a1a" Oct 30 13:29:37.946108 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:37.946087 7010 scope.go:122] "RemoveContainer" containerID="6d7e784beb25ef5803d3587e7e3992460f6357cbaae9f0085f8129152a489a1a" Oct 30 13:29:37.946421 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:29:37.946398 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d7e784beb25ef5803d3587e7e3992460f6357cbaae9f0085f8129152a489a1a\": container with ID starting with 6d7e784beb25ef5803d3587e7e3992460f6357cbaae9f0085f8129152a489a1a not found: ID does not exist" containerID="6d7e784beb25ef5803d3587e7e3992460f6357cbaae9f0085f8129152a489a1a" Oct 30 13:29:37.946524 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:37.946434 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d7e784beb25ef5803d3587e7e3992460f6357cbaae9f0085f8129152a489a1a"} err="failed to get container status \"6d7e784beb25ef5803d3587e7e3992460f6357cbaae9f0085f8129152a489a1a\": rpc error: code = NotFound desc = could not find container \"6d7e784beb25ef5803d3587e7e3992460f6357cbaae9f0085f8129152a489a1a\": container with ID starting with 6d7e784beb25ef5803d3587e7e3992460f6357cbaae9f0085f8129152a489a1a not found: ID does not exist" Oct 30 13:29:37.996916 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:37.996883 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm"] Oct 30 13:29:38.001296 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:38.001261 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-376c89-56cd6c8554-6flzm"] Oct 30 13:29:39.489335 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:39.489303 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c819c5-dff1-4b26-a66f-d4d2f6376387" path="/var/lib/kubelet/pods/d1c819c5-dff1-4b26-a66f-d4d2f6376387/volumes" Oct 30 13:29:44.651528 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.651478 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg"] Oct 30 13:29:44.652061 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.652048 7010 cpu_manager.go:414] "containerMap: removing container" podUID="d1c819c5-dff1-4b26-a66f-d4d2f6376387" containerName="payload-processing" Oct 30 13:29:44.652061 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.652062 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c819c5-dff1-4b26-a66f-d4d2f6376387" containerName="payload-processing" Oct 30 13:29:44.652146 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.652080 7010 cpu_manager.go:414] "containerMap: removing container" podUID="be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a" containerName="maas-api" Oct 30 13:29:44.652146 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.652086 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a" containerName="maas-api" Oct 30 13:29:44.652146 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.652105 7010 cpu_manager.go:414] "containerMap: removing container" podUID="03a68869-2b11-4796-a767-ab7402efbf87" containerName="istio-proxy" Oct 30 13:29:44.652146 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.652112 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a68869-2b11-4796-a767-ab7402efbf87" containerName="istio-proxy" Oct 30 13:29:44.652146 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.652140 7010 cpu_manager.go:414] "containerMap: removing container" podUID="353779fe-c229-48e5-b17b-095bd6c32f59" containerName="payload-processing" Oct 30 13:29:44.652146 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.652144 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353779fe-c229-48e5-b17b-095bd6c32f59" containerName="payload-processing" Oct 30 13:29:44.652327 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.652219 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a68869-2b11-4796-a767-ab7402efbf87" containerName="istio-proxy" Oct 30 13:29:44.652327 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.652233 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="353779fe-c229-48e5-b17b-095bd6c32f59" containerName="payload-processing" Oct 30 13:29:44.652327 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.652241 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c819c5-dff1-4b26-a66f-d4d2f6376387" containerName="payload-processing" Oct 30 13:29:44.652327 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.652249 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="be8b97ee-4a2c-4987-9dbe-3b1ccc0f423a" containerName="maas-api" Oct 30 13:29:44.656887 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.656871 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:29:44.686507 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.686450 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" podUID="7459ef64-d4fa-4130-acd8-4a0162654b1d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:29:44.701038 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.700979 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-4b0ea9-serving-cert\"" Oct 30 13:29:44.705566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.705533 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg"] Oct 30 13:29:44.765837 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.765802 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg\" (UID: \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:29:44.766050 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.766026 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv9pk\" (UniqueName: \"kubernetes.io/projected/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-kube-api-access-fv9pk\") pod \"maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg\" (UID: \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:29:44.766103 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.766076 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg\" (UID: \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:29:44.766174 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.766160 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg\" (UID: \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:29:44.867723 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.867687 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv9pk\" (UniqueName: \"kubernetes.io/projected/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-kube-api-access-fv9pk\") pod \"maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg\" (UID: \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:29:44.867922 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.867741 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg\" (UID: \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:29:44.867922 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.867777 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg\" (UID: \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:29:44.867922 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.867857 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg\" (UID: \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:29:44.868558 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.868527 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg\" (UID: \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:29:44.870267 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.870238 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg\" (UID: \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:29:44.870422 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.870402 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg\" (UID: \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:29:44.891753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.891729 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv9pk\" (UniqueName: \"kubernetes.io/projected/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-kube-api-access-fv9pk\") pod \"maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg\" (UID: \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:29:44.913101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.913035 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" podUID="ef58e88a-0bad-4099-9194-d4cd16997478" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:29:44.970657 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:44.970620 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:29:45.293817 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:45.293792 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg"] Oct 30 13:29:45.295767 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:29:45.295741 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8430e34_fc9c_42de_82d5_b4d2ffd07dfc.slice/crio-9963517cff3e752490b24de99174e871f6ced703c1d8f5430d5785e8528b810a WatchSource:0}: Error finding container 9963517cff3e752490b24de99174e871f6ced703c1d8f5430d5785e8528b810a: Status 404 returned error can't find the container with id 9963517cff3e752490b24de99174e871f6ced703c1d8f5430d5785e8528b810a Oct 30 13:29:45.297203 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:45.297186 7010 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:29:46.001589 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:45.999886 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" event={"ID":"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc","Type":"ContainerStarted","Data":"b62978e278df850f59eec426bf3e0200747b8d5951f08c0fa49c8bc3714e9401"} Oct 30 13:29:46.001589 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:45.999998 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" event={"ID":"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc","Type":"ContainerStarted","Data":"9963517cff3e752490b24de99174e871f6ced703c1d8f5430d5785e8528b810a"} Oct 30 13:29:46.001589 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:46.000324 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:29:47.912500 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:47.912467 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" Oct 30 13:29:47.956655 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:47.956600 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" podStartSLOduration=3.616963635 podStartE2EDuration="3.956586012s" totalImagesPullingTime="339.622377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:46.06074782 +0000 UTC m=+2242.692934984" watchObservedRunningTime="2026-10-30 13:29:47.956586012 +0000 UTC m=+2244.588773220" Oct 30 13:29:48.012515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:48.012455 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7459ef64-d4fa-4130-acd8-4a0162654b1d-kube-api-access-mmpq8\" (UniqueName: \"kubernetes.io/projected/7459ef64-d4fa-4130-acd8-4a0162654b1d-kube-api-access-mmpq8\") pod \"7459ef64-d4fa-4130-acd8-4a0162654b1d\" (UID: \"7459ef64-d4fa-4130-acd8-4a0162654b1d\") " Oct 30 13:29:48.012733 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:48.012692 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7459ef64-d4fa-4130-acd8-4a0162654b1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7459ef64-d4fa-4130-acd8-4a0162654b1d-plugins-config-volume\") pod \"7459ef64-d4fa-4130-acd8-4a0162654b1d\" (UID: \"7459ef64-d4fa-4130-acd8-4a0162654b1d\") " Oct 30 13:29:48.013131 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:48.013102 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7459ef64-d4fa-4130-acd8-4a0162654b1d-plugins-config-volume" pod "7459ef64-d4fa-4130-acd8-4a0162654b1d" (UID: "7459ef64-d4fa-4130-acd8-4a0162654b1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:48.013372 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:48.013343 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7459ef64-d4fa-4130-acd8-4a0162654b1d-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:48.015551 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:48.015526 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7459ef64-d4fa-4130-acd8-4a0162654b1d-kube-api-access-mmpq8" pod "7459ef64-d4fa-4130-acd8-4a0162654b1d" (UID: "7459ef64-d4fa-4130-acd8-4a0162654b1d"). InnerVolumeSpecName "kube-api-access-mmpq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:48.016569 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:48.016538 7010 generic.go:356] "Generic (PLEG): container finished" podID="7459ef64-d4fa-4130-acd8-4a0162654b1d" containerID="b7ccb5d3b14ad984fc68fcdb5cf57281abec7877e79bc560ac546d601c38630e" exitCode=0 Oct 30 13:29:48.016775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:48.016590 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" Oct 30 13:29:48.016861 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:48.016582 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" event={"ID":"7459ef64-d4fa-4130-acd8-4a0162654b1d","Type":"ContainerDied","Data":"b7ccb5d3b14ad984fc68fcdb5cf57281abec7877e79bc560ac546d601c38630e"} Oct 30 13:29:48.016861 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:48.016854 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk" event={"ID":"7459ef64-d4fa-4130-acd8-4a0162654b1d","Type":"ContainerDied","Data":"d8ef1fe062ab5323537588ab20a67f5437d0a9040ffb2ae58e68c3c648f43994"} Oct 30 13:29:48.016983 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:48.016888 7010 scope.go:122] "RemoveContainer" containerID="b7ccb5d3b14ad984fc68fcdb5cf57281abec7877e79bc560ac546d601c38630e" Oct 30 13:29:48.036120 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:48.036104 7010 scope.go:122] "RemoveContainer" containerID="b7ccb5d3b14ad984fc68fcdb5cf57281abec7877e79bc560ac546d601c38630e" Oct 30 13:29:48.036403 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:29:48.036382 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7ccb5d3b14ad984fc68fcdb5cf57281abec7877e79bc560ac546d601c38630e\": container with ID starting with b7ccb5d3b14ad984fc68fcdb5cf57281abec7877e79bc560ac546d601c38630e not found: ID does not exist" containerID="b7ccb5d3b14ad984fc68fcdb5cf57281abec7877e79bc560ac546d601c38630e" Oct 30 13:29:48.036458 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:48.036412 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7ccb5d3b14ad984fc68fcdb5cf57281abec7877e79bc560ac546d601c38630e"} err="failed to get container status \"b7ccb5d3b14ad984fc68fcdb5cf57281abec7877e79bc560ac546d601c38630e\": rpc error: code = NotFound desc = could not find container \"b7ccb5d3b14ad984fc68fcdb5cf57281abec7877e79bc560ac546d601c38630e\": container with ID starting with b7ccb5d3b14ad984fc68fcdb5cf57281abec7877e79bc560ac546d601c38630e not found: ID does not exist" Oct 30 13:29:48.070992 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:48.070956 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk"] Oct 30 13:29:48.075265 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:48.075236 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5c7ba3-59b78cf6dc-chlxk"] Oct 30 13:29:48.114649 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:48.114614 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmpq8\" (UniqueName: \"kubernetes.io/projected/7459ef64-d4fa-4130-acd8-4a0162654b1d-kube-api-access-mmpq8\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:49.489935 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:49.489903 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7459ef64-d4fa-4130-acd8-4a0162654b1d" path="/var/lib/kubelet/pods/7459ef64-d4fa-4130-acd8-4a0162654b1d/volumes" Oct 30 13:29:52.013307 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:52.013269 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:29:54.913167 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:54.913126 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" podUID="ef58e88a-0bad-4099-9194-d4cd16997478" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:29:57.360776 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:57.360738 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" Oct 30 13:29:57.463297 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:57.463252 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef58e88a-0bad-4099-9194-d4cd16997478-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef58e88a-0bad-4099-9194-d4cd16997478-plugins-config-volume\") pod \"ef58e88a-0bad-4099-9194-d4cd16997478\" (UID: \"ef58e88a-0bad-4099-9194-d4cd16997478\") " Oct 30 13:29:57.463297 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:57.463304 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef58e88a-0bad-4099-9194-d4cd16997478-kube-api-access-fdssr\" (UniqueName: \"kubernetes.io/projected/ef58e88a-0bad-4099-9194-d4cd16997478-kube-api-access-fdssr\") pod \"ef58e88a-0bad-4099-9194-d4cd16997478\" (UID: \"ef58e88a-0bad-4099-9194-d4cd16997478\") " Oct 30 13:29:57.463754 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:57.463722 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef58e88a-0bad-4099-9194-d4cd16997478-plugins-config-volume" pod "ef58e88a-0bad-4099-9194-d4cd16997478" (UID: "ef58e88a-0bad-4099-9194-d4cd16997478"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:29:57.463937 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:57.463921 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef58e88a-0bad-4099-9194-d4cd16997478-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:57.465659 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:57.465632 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef58e88a-0bad-4099-9194-d4cd16997478-kube-api-access-fdssr" pod "ef58e88a-0bad-4099-9194-d4cd16997478" (UID: "ef58e88a-0bad-4099-9194-d4cd16997478"). InnerVolumeSpecName "kube-api-access-fdssr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:29:57.565000 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:57.564961 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdssr\" (UniqueName: \"kubernetes.io/projected/ef58e88a-0bad-4099-9194-d4cd16997478-kube-api-access-fdssr\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:29:58.086583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:58.086543 7010 generic.go:356] "Generic (PLEG): container finished" podID="ef58e88a-0bad-4099-9194-d4cd16997478" containerID="460db5f17ee38a8decce8c834e64196aac68f83abb4881848d4ab63c4361dba4" exitCode=0 Oct 30 13:29:58.086798 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:58.086630 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" Oct 30 13:29:58.086798 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:58.086656 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" event={"ID":"ef58e88a-0bad-4099-9194-d4cd16997478","Type":"ContainerDied","Data":"460db5f17ee38a8decce8c834e64196aac68f83abb4881848d4ab63c4361dba4"} Oct 30 13:29:58.086798 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:58.086717 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws" event={"ID":"ef58e88a-0bad-4099-9194-d4cd16997478","Type":"ContainerDied","Data":"99713d981dd913ccbfc89d070860beeb4bfb1b85d33fa837ba55d4c7588d5fa2"} Oct 30 13:29:58.086798 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:58.086744 7010 scope.go:122] "RemoveContainer" containerID="460db5f17ee38a8decce8c834e64196aac68f83abb4881848d4ab63c4361dba4" Oct 30 13:29:58.096567 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:58.096548 7010 scope.go:122] "RemoveContainer" containerID="460db5f17ee38a8decce8c834e64196aac68f83abb4881848d4ab63c4361dba4" Oct 30 13:29:58.096866 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:29:58.096842 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460db5f17ee38a8decce8c834e64196aac68f83abb4881848d4ab63c4361dba4\": container with ID starting with 460db5f17ee38a8decce8c834e64196aac68f83abb4881848d4ab63c4361dba4 not found: ID does not exist" containerID="460db5f17ee38a8decce8c834e64196aac68f83abb4881848d4ab63c4361dba4" Oct 30 13:29:58.096918 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:58.096874 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460db5f17ee38a8decce8c834e64196aac68f83abb4881848d4ab63c4361dba4"} err="failed to get container status \"460db5f17ee38a8decce8c834e64196aac68f83abb4881848d4ab63c4361dba4\": rpc error: code = NotFound desc = could not find container \"460db5f17ee38a8decce8c834e64196aac68f83abb4881848d4ab63c4361dba4\": container with ID starting with 460db5f17ee38a8decce8c834e64196aac68f83abb4881848d4ab63c4361dba4 not found: ID does not exist" Oct 30 13:29:58.140756 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:58.140726 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws"] Oct 30 13:29:58.153141 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:58.153112 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-376c89-98dd6d4cf-qrnws"] Oct 30 13:29:59.401824 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:29:59.401786 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:29:59.489785 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:29:59.489748 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef58e88a-0bad-4099-9194-d4cd16997478" path="/var/lib/kubelet/pods/ef58e88a-0bad-4099-9194-d4cd16997478/volumes" Oct 30 13:30:00.155380 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.155339 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb"] Oct 30 13:30:00.155968 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.155950 7010 cpu_manager.go:414] "containerMap: removing container" podUID="7459ef64-d4fa-4130-acd8-4a0162654b1d" containerName="payload-pre-processing" Oct 30 13:30:00.156021 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.155969 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7459ef64-d4fa-4130-acd8-4a0162654b1d" containerName="payload-pre-processing" Oct 30 13:30:00.156021 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.156008 7010 cpu_manager.go:414] "containerMap: removing container" podUID="ef58e88a-0bad-4099-9194-d4cd16997478" containerName="payload-pre-processing" Oct 30 13:30:00.156021 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.156014 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef58e88a-0bad-4099-9194-d4cd16997478" containerName="payload-pre-processing" Oct 30 13:30:00.156121 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.156097 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="7459ef64-d4fa-4130-acd8-4a0162654b1d" containerName="payload-pre-processing" Oct 30 13:30:00.156121 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.156113 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef58e88a-0bad-4099-9194-d4cd16997478" containerName="payload-pre-processing" Oct 30 13:30:00.161298 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.161276 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" Oct 30 13:30:00.186550 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.186480 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:30:00.214024 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.213969 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb"] Oct 30 13:30:00.219373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.219338 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh"] Oct 30 13:30:00.224587 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.224556 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" Oct 30 13:30:00.247967 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.247933 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh"] Oct 30 13:30:00.254059 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.252578 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9"] Oct 30 13:30:00.258669 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.258629 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j"] Oct 30 13:30:00.258880 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.258756 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" Oct 30 13:30:00.264624 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.264594 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" Oct 30 13:30:00.265084 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.265058 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9"] Oct 30 13:30:00.280312 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.278894 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j"] Oct 30 13:30:00.290137 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.290104 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk"] Oct 30 13:30:00.294888 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.294859 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8"] Oct 30 13:30:00.295163 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.295140 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" Oct 30 13:30:00.307757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.307732 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" Oct 30 13:30:00.321194 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.316636 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7779049-af76-4965-8838-00e5c1eda794-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb\" (UID: \"d7779049-af76-4965-8838-00e5c1eda794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" Oct 30 13:30:00.321194 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.316700 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh7xc\" (UniqueName: \"kubernetes.io/projected/d7779049-af76-4965-8838-00e5c1eda794-kube-api-access-vh7xc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb\" (UID: \"d7779049-af76-4965-8838-00e5c1eda794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" Oct 30 13:30:00.335017 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.334872 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk"] Oct 30 13:30:00.340158 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.340121 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8"] Oct 30 13:30:00.418050 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.417953 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd57d65-b68c-48f4-b173-827652241344-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9\" (UID: \"6dd57d65-b68c-48f4-b173-827652241344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" Oct 30 13:30:00.418050 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.418021 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd4r6\" (UniqueName: \"kubernetes.io/projected/6dd57d65-b68c-48f4-b173-827652241344-kube-api-access-sd4r6\") pod \"maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9\" (UID: \"6dd57d65-b68c-48f4-b173-827652241344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" Oct 30 13:30:00.418521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.418050 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/998f2f9e-5b70-42d1-a7ed-66de6f1c9146-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk\" (UID: \"998f2f9e-5b70-42d1-a7ed-66de6f1c9146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" Oct 30 13:30:00.418521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.418084 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrmtx\" (UniqueName: \"kubernetes.io/projected/998f2f9e-5b70-42d1-a7ed-66de6f1c9146-kube-api-access-hrmtx\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk\" (UID: \"998f2f9e-5b70-42d1-a7ed-66de6f1c9146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" Oct 30 13:30:00.418521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.418175 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b1ca04-4bf9-48c5-abe8-644a12a66127-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh\" (UID: \"38b1ca04-4bf9-48c5-abe8-644a12a66127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" Oct 30 13:30:00.418521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.418236 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3d7096-56e6-4491-a5f1-737dcb106b1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8\" (UID: \"6b3d7096-56e6-4491-a5f1-737dcb106b1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" Oct 30 13:30:00.418521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.418287 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjdxs\" (UniqueName: \"kubernetes.io/projected/6b3d7096-56e6-4491-a5f1-737dcb106b1d-kube-api-access-sjdxs\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8\" (UID: \"6b3d7096-56e6-4491-a5f1-737dcb106b1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" Oct 30 13:30:00.418521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.418334 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7ea0eb-4b45-4168-9df5-2a6591edd74d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j\" (UID: \"6b7ea0eb-4b45-4168-9df5-2a6591edd74d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" Oct 30 13:30:00.418521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.418364 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7779049-af76-4965-8838-00e5c1eda794-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb\" (UID: \"d7779049-af76-4965-8838-00e5c1eda794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" Oct 30 13:30:00.418521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.418473 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7xc\" (UniqueName: \"kubernetes.io/projected/d7779049-af76-4965-8838-00e5c1eda794-kube-api-access-vh7xc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb\" (UID: \"d7779049-af76-4965-8838-00e5c1eda794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" Oct 30 13:30:00.418825 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.418574 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbtfb\" (UniqueName: \"kubernetes.io/projected/6b7ea0eb-4b45-4168-9df5-2a6591edd74d-kube-api-access-hbtfb\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j\" (UID: \"6b7ea0eb-4b45-4168-9df5-2a6591edd74d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" Oct 30 13:30:00.418825 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.418630 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqrj4\" (UniqueName: \"kubernetes.io/projected/38b1ca04-4bf9-48c5-abe8-644a12a66127-kube-api-access-rqrj4\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh\" (UID: \"38b1ca04-4bf9-48c5-abe8-644a12a66127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" Oct 30 13:30:00.419071 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.419054 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7779049-af76-4965-8838-00e5c1eda794-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb\" (UID: \"d7779049-af76-4965-8838-00e5c1eda794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" Oct 30 13:30:00.466058 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.466026 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh7xc\" (UniqueName: \"kubernetes.io/projected/d7779049-af76-4965-8838-00e5c1eda794-kube-api-access-vh7xc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb\" (UID: \"d7779049-af76-4965-8838-00e5c1eda794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" Oct 30 13:30:00.476599 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.476558 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" Oct 30 13:30:00.520082 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.520044 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd4r6\" (UniqueName: \"kubernetes.io/projected/6dd57d65-b68c-48f4-b173-827652241344-kube-api-access-sd4r6\") pod \"maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9\" (UID: \"6dd57d65-b68c-48f4-b173-827652241344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" Oct 30 13:30:00.520279 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.520088 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/998f2f9e-5b70-42d1-a7ed-66de6f1c9146-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk\" (UID: \"998f2f9e-5b70-42d1-a7ed-66de6f1c9146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" Oct 30 13:30:00.520279 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.520201 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrmtx\" (UniqueName: \"kubernetes.io/projected/998f2f9e-5b70-42d1-a7ed-66de6f1c9146-kube-api-access-hrmtx\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk\" (UID: \"998f2f9e-5b70-42d1-a7ed-66de6f1c9146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" Oct 30 13:30:00.520279 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.520254 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b1ca04-4bf9-48c5-abe8-644a12a66127-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh\" (UID: \"38b1ca04-4bf9-48c5-abe8-644a12a66127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" Oct 30 13:30:00.520451 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.520293 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3d7096-56e6-4491-a5f1-737dcb106b1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8\" (UID: \"6b3d7096-56e6-4491-a5f1-737dcb106b1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" Oct 30 13:30:00.520451 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.520362 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjdxs\" (UniqueName: \"kubernetes.io/projected/6b3d7096-56e6-4491-a5f1-737dcb106b1d-kube-api-access-sjdxs\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8\" (UID: \"6b3d7096-56e6-4491-a5f1-737dcb106b1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" Oct 30 13:30:00.520451 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.520427 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7ea0eb-4b45-4168-9df5-2a6591edd74d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j\" (UID: \"6b7ea0eb-4b45-4168-9df5-2a6591edd74d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" Oct 30 13:30:00.520650 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.520623 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbtfb\" (UniqueName: \"kubernetes.io/projected/6b7ea0eb-4b45-4168-9df5-2a6591edd74d-kube-api-access-hbtfb\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j\" (UID: \"6b7ea0eb-4b45-4168-9df5-2a6591edd74d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" Oct 30 13:30:00.520712 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.520700 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqrj4\" (UniqueName: \"kubernetes.io/projected/38b1ca04-4bf9-48c5-abe8-644a12a66127-kube-api-access-rqrj4\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh\" (UID: \"38b1ca04-4bf9-48c5-abe8-644a12a66127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" Oct 30 13:30:00.520900 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.520877 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd57d65-b68c-48f4-b173-827652241344-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9\" (UID: \"6dd57d65-b68c-48f4-b173-827652241344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" Oct 30 13:30:00.521121 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.521095 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/998f2f9e-5b70-42d1-a7ed-66de6f1c9146-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk\" (UID: \"998f2f9e-5b70-42d1-a7ed-66de6f1c9146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" Oct 30 13:30:00.521191 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.521153 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7ea0eb-4b45-4168-9df5-2a6591edd74d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j\" (UID: \"6b7ea0eb-4b45-4168-9df5-2a6591edd74d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" Oct 30 13:30:00.521191 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.521176 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3d7096-56e6-4491-a5f1-737dcb106b1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8\" (UID: \"6b3d7096-56e6-4491-a5f1-737dcb106b1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" Oct 30 13:30:00.521752 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.521730 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd57d65-b68c-48f4-b173-827652241344-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9\" (UID: \"6dd57d65-b68c-48f4-b173-827652241344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" Oct 30 13:30:00.521818 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.521730 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b1ca04-4bf9-48c5-abe8-644a12a66127-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh\" (UID: \"38b1ca04-4bf9-48c5-abe8-644a12a66127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" Oct 30 13:30:00.588198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.584051 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqrj4\" (UniqueName: \"kubernetes.io/projected/38b1ca04-4bf9-48c5-abe8-644a12a66127-kube-api-access-rqrj4\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh\" (UID: \"38b1ca04-4bf9-48c5-abe8-644a12a66127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" Oct 30 13:30:00.588198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.584786 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjdxs\" (UniqueName: \"kubernetes.io/projected/6b3d7096-56e6-4491-a5f1-737dcb106b1d-kube-api-access-sjdxs\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8\" (UID: \"6b3d7096-56e6-4491-a5f1-737dcb106b1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" Oct 30 13:30:00.588198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.585367 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd4r6\" (UniqueName: \"kubernetes.io/projected/6dd57d65-b68c-48f4-b173-827652241344-kube-api-access-sd4r6\") pod \"maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9\" (UID: \"6dd57d65-b68c-48f4-b173-827652241344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" Oct 30 13:30:00.588198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.587946 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbtfb\" (UniqueName: \"kubernetes.io/projected/6b7ea0eb-4b45-4168-9df5-2a6591edd74d-kube-api-access-hbtfb\") pod \"maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j\" (UID: \"6b7ea0eb-4b45-4168-9df5-2a6591edd74d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" Oct 30 13:30:00.595863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.591545 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrmtx\" (UniqueName: \"kubernetes.io/projected/998f2f9e-5b70-42d1-a7ed-66de6f1c9146-kube-api-access-hrmtx\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk\" (UID: \"998f2f9e-5b70-42d1-a7ed-66de6f1c9146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" Oct 30 13:30:00.596027 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.595986 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" Oct 30 13:30:00.599234 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.599210 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" Oct 30 13:30:00.649907 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.624745 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" Oct 30 13:30:00.649907 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.649137 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" Oct 30 13:30:00.859534 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.859265 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" Oct 30 13:30:00.860041 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:00.859862 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb"] Oct 30 13:30:01.118527 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:01.115676 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9"] Oct 30 13:30:01.128676 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:01.128544 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" event={"ID":"d7779049-af76-4965-8838-00e5c1eda794","Type":"ContainerStarted","Data":"370ddbe0e07eba22779c942ad25541a59f8b8f1530ce6023cb514e622dc369d0"} Oct 30 13:30:01.199558 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:01.197311 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8"] Oct 30 13:30:01.278105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:01.278075 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk"] Oct 30 13:30:01.280672 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:01.280643 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j"] Oct 30 13:30:01.299237 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:30:01.299210 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998f2f9e_5b70_42d1_a7ed_66de6f1c9146.slice/crio-cb8f83e23736ad671ffbe79691d39e493d595634e11f59c6977baaee71a224b5 WatchSource:0}: Error finding container cb8f83e23736ad671ffbe79691d39e493d595634e11f59c6977baaee71a224b5: Status 404 returned error can't find the container with id cb8f83e23736ad671ffbe79691d39e493d595634e11f59c6977baaee71a224b5 Oct 30 13:30:01.299648 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:30:01.299624 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3d7096_56e6_4491_a5f1_737dcb106b1d.slice/crio-8c6f737e33d70ec9272c5f21844cf09d1c9fde9ce39f7f93a8a4fcd5e214af52 WatchSource:0}: Error finding container 8c6f737e33d70ec9272c5f21844cf09d1c9fde9ce39f7f93a8a4fcd5e214af52: Status 404 returned error can't find the container with id 8c6f737e33d70ec9272c5f21844cf09d1c9fde9ce39f7f93a8a4fcd5e214af52 Oct 30 13:30:01.301080 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:30:01.301058 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7ea0eb_4b45_4168_9df5_2a6591edd74d.slice/crio-08be9a5a56ead25cd25970759d18ee224f92cf83acfeedb35c3d2f74bf57339d WatchSource:0}: Error finding container 08be9a5a56ead25cd25970759d18ee224f92cf83acfeedb35c3d2f74bf57339d: Status 404 returned error can't find the container with id 08be9a5a56ead25cd25970759d18ee224f92cf83acfeedb35c3d2f74bf57339d Oct 30 13:30:01.345124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:01.345096 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh"] Oct 30 13:30:02.139448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.139401 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" event={"ID":"d7779049-af76-4965-8838-00e5c1eda794","Type":"ContainerStarted","Data":"8b58f87a68698bedc7fbd200cdbdc188ebcab863314448f5b98aa168d3a8c010"} Oct 30 13:30:02.141909 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.141805 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" event={"ID":"998f2f9e-5b70-42d1-a7ed-66de6f1c9146","Type":"ContainerStarted","Data":"6fff11195a0a41fb7d73e4b5c655b6d7674179dc28e93dd868f86421ec83221e"} Oct 30 13:30:02.141909 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.141872 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" event={"ID":"998f2f9e-5b70-42d1-a7ed-66de6f1c9146","Type":"ContainerStarted","Data":"cb8f83e23736ad671ffbe79691d39e493d595634e11f59c6977baaee71a224b5"} Oct 30 13:30:02.144004 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.143963 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" event={"ID":"6b7ea0eb-4b45-4168-9df5-2a6591edd74d","Type":"ContainerStarted","Data":"d4788db39907165e3187436901ad7f8e76c091230462fefedd5830a184a97edf"} Oct 30 13:30:02.144182 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.144023 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" event={"ID":"6b7ea0eb-4b45-4168-9df5-2a6591edd74d","Type":"ContainerStarted","Data":"08be9a5a56ead25cd25970759d18ee224f92cf83acfeedb35c3d2f74bf57339d"} Oct 30 13:30:02.146725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.146690 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" event={"ID":"6b3d7096-56e6-4491-a5f1-737dcb106b1d","Type":"ContainerStarted","Data":"ff2fc0f54a393712d2bd57d797bc0635e958a16d0b8743c52be01b06b2044d4d"} Oct 30 13:30:02.146913 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.146750 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" event={"ID":"6b3d7096-56e6-4491-a5f1-737dcb106b1d","Type":"ContainerStarted","Data":"8c6f737e33d70ec9272c5f21844cf09d1c9fde9ce39f7f93a8a4fcd5e214af52"} Oct 30 13:30:02.148749 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.148711 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" event={"ID":"6dd57d65-b68c-48f4-b173-827652241344","Type":"ContainerStarted","Data":"fbf132faae50360f33ac4fb21a66bce32b9cedef2a0a5a11650fab7530806a14"} Oct 30 13:30:02.148924 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.148773 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" event={"ID":"6dd57d65-b68c-48f4-b173-827652241344","Type":"ContainerStarted","Data":"b62be3228dfda0f6fcb951a1afb81d2d30f1100b65758b03125f975d171882fc"} Oct 30 13:30:02.150549 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.150473 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" event={"ID":"38b1ca04-4bf9-48c5-abe8-644a12a66127","Type":"ContainerStarted","Data":"0e22329b87b19e0e7bc801e55791965a70f4b5577e5d337b77ad648cc401142e"} Oct 30 13:30:02.150709 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.150566 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" event={"ID":"38b1ca04-4bf9-48c5-abe8-644a12a66127","Type":"ContainerStarted","Data":"79540cf5918ae9f949375456274dfbeee6c8da6e03c9ae0ec311d73886c508e8"} Oct 30 13:30:02.252517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.240771 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" podStartSLOduration=2.240748493 podStartE2EDuration="2.240748493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:02.217196048 +0000 UTC m=+2258.849383226" watchObservedRunningTime="2026-10-30 13:30:02.240748493 +0000 UTC m=+2258.872935656" Oct 30 13:30:02.261761 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.260480 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" podStartSLOduration=2.260462904 podStartE2EDuration="2.260462904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:02.259553834 +0000 UTC m=+2258.891741048" watchObservedRunningTime="2026-10-30 13:30:02.260462904 +0000 UTC m=+2258.892650066" Oct 30 13:30:02.355705 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.355647 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" podStartSLOduration=2.355631347 podStartE2EDuration="2.355631347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:02.308319741 +0000 UTC m=+2258.940506907" watchObservedRunningTime="2026-10-30 13:30:02.355631347 +0000 UTC m=+2258.987818508" Oct 30 13:30:02.355949 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.355920 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" podStartSLOduration=2.355910967 podStartE2EDuration="2.355910967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:02.352425744 +0000 UTC m=+2258.984612908" watchObservedRunningTime="2026-10-30 13:30:02.355910967 +0000 UTC m=+2258.988098130" Oct 30 13:30:02.444077 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.444018 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" podStartSLOduration=2.44400288 podStartE2EDuration="2.44400288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:02.441941591 +0000 UTC m=+2259.074128754" watchObservedRunningTime="2026-10-30 13:30:02.44400288 +0000 UTC m=+2259.076190042" Oct 30 13:30:02.444396 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.444374 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" podStartSLOduration=2.444366558 podStartE2EDuration="2.444366558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:02.39773993 +0000 UTC m=+2259.029927095" watchObservedRunningTime="2026-10-30 13:30:02.444366558 +0000 UTC m=+2259.076553719" Oct 30 13:30:02.677841 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.677754 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw"] Oct 30 13:30:02.682077 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.682054 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.713534 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.707677 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-fdd4d4-openshift-default-dockercfg-2qk7r\"" Oct 30 13:30:02.713534 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.710834 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw"] Oct 30 13:30:02.827532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.824366 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istiod-ca-cert\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.827532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.824586 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-data\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.827532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.824631 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-envoy\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.827532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.824716 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-credential-socket\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.827532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.824769 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-workload-certs\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.827532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.824912 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-workload-socket\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.827532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.824943 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btz5g\" (UniqueName: \"kubernetes.io/projected/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-kube-api-access-btz5g\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.827532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.824972 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-token\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.827532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.825061 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-podinfo\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.929506 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.929379 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-workload-socket\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.929833 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.929806 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btz5g\" (UniqueName: \"kubernetes.io/projected/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-kube-api-access-btz5g\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.929994 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.929976 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-token\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.930163 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.930147 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-podinfo\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.930313 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.930298 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istiod-ca-cert\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.930464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.930450 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-data\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.930602 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.930587 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-envoy\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.930761 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.930747 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-credential-socket\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.930894 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.930877 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-workload-certs\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.931582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.931559 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-workload-certs\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.938023 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.937986 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-workload-socket\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.939461 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.939434 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-data\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.944602 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.944100 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istiod-ca-cert\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.944756 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.944257 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-credential-socket\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.966366 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.966331 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-podinfo\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.966980 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.966945 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-envoy\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.977447 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.977372 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-token\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:02.979243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:02.978239 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btz5g\" (UniqueName: \"kubernetes.io/projected/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-kube-api-access-btz5g\") pod \"e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:03.000522 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.000457 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:03.161938 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.161901 7010 generic.go:356] "Generic (PLEG): container finished" podID="6b3d7096-56e6-4491-a5f1-737dcb106b1d" containerID="ff2fc0f54a393712d2bd57d797bc0635e958a16d0b8743c52be01b06b2044d4d" exitCode=60 Oct 30 13:30:03.162647 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.162341 7010 scope.go:122] "RemoveContainer" containerID="ff2fc0f54a393712d2bd57d797bc0635e958a16d0b8743c52be01b06b2044d4d" Oct 30 13:30:03.162935 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.162636 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" event={"ID":"6b3d7096-56e6-4491-a5f1-737dcb106b1d","Type":"ContainerDied","Data":"ff2fc0f54a393712d2bd57d797bc0635e958a16d0b8743c52be01b06b2044d4d"} Oct 30 13:30:03.185540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.184628 7010 generic.go:356] "Generic (PLEG): container finished" podID="6dd57d65-b68c-48f4-b173-827652241344" containerID="fbf132faae50360f33ac4fb21a66bce32b9cedef2a0a5a11650fab7530806a14" exitCode=60 Oct 30 13:30:03.185540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.184737 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" event={"ID":"6dd57d65-b68c-48f4-b173-827652241344","Type":"ContainerDied","Data":"fbf132faae50360f33ac4fb21a66bce32b9cedef2a0a5a11650fab7530806a14"} Oct 30 13:30:03.185540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.185244 7010 scope.go:122] "RemoveContainer" containerID="fbf132faae50360f33ac4fb21a66bce32b9cedef2a0a5a11650fab7530806a14" Oct 30 13:30:03.203796 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.194861 7010 generic.go:356] "Generic (PLEG): container finished" podID="38b1ca04-4bf9-48c5-abe8-644a12a66127" containerID="0e22329b87b19e0e7bc801e55791965a70f4b5577e5d337b77ad648cc401142e" exitCode=60 Oct 30 13:30:03.203796 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.194962 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" event={"ID":"38b1ca04-4bf9-48c5-abe8-644a12a66127","Type":"ContainerDied","Data":"0e22329b87b19e0e7bc801e55791965a70f4b5577e5d337b77ad648cc401142e"} Oct 30 13:30:03.203796 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.195392 7010 scope.go:122] "RemoveContainer" containerID="0e22329b87b19e0e7bc801e55791965a70f4b5577e5d337b77ad648cc401142e" Oct 30 13:30:03.203796 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.197790 7010 generic.go:356] "Generic (PLEG): container finished" podID="d7779049-af76-4965-8838-00e5c1eda794" containerID="8b58f87a68698bedc7fbd200cdbdc188ebcab863314448f5b98aa168d3a8c010" exitCode=60 Oct 30 13:30:03.203796 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.197954 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" event={"ID":"d7779049-af76-4965-8838-00e5c1eda794","Type":"ContainerDied","Data":"8b58f87a68698bedc7fbd200cdbdc188ebcab863314448f5b98aa168d3a8c010"} Oct 30 13:30:03.203796 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.198373 7010 scope.go:122] "RemoveContainer" containerID="8b58f87a68698bedc7fbd200cdbdc188ebcab863314448f5b98aa168d3a8c010" Oct 30 13:30:03.207929 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.207880 7010 generic.go:356] "Generic (PLEG): container finished" podID="998f2f9e-5b70-42d1-a7ed-66de6f1c9146" containerID="6fff11195a0a41fb7d73e4b5c655b6d7674179dc28e93dd868f86421ec83221e" exitCode=60 Oct 30 13:30:03.208043 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.208022 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" event={"ID":"998f2f9e-5b70-42d1-a7ed-66de6f1c9146","Type":"ContainerDied","Data":"6fff11195a0a41fb7d73e4b5c655b6d7674179dc28e93dd868f86421ec83221e"} Oct 30 13:30:03.208279 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.208256 7010 scope.go:122] "RemoveContainer" containerID="6fff11195a0a41fb7d73e4b5c655b6d7674179dc28e93dd868f86421ec83221e" Oct 30 13:30:03.213287 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.213260 7010 generic.go:356] "Generic (PLEG): container finished" podID="6b7ea0eb-4b45-4168-9df5-2a6591edd74d" containerID="d4788db39907165e3187436901ad7f8e76c091230462fefedd5830a184a97edf" exitCode=60 Oct 30 13:30:03.213385 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.213368 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" event={"ID":"6b7ea0eb-4b45-4168-9df5-2a6591edd74d","Type":"ContainerDied","Data":"d4788db39907165e3187436901ad7f8e76c091230462fefedd5830a184a97edf"} Oct 30 13:30:03.213824 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.213801 7010 scope.go:122] "RemoveContainer" containerID="d4788db39907165e3187436901ad7f8e76c091230462fefedd5830a184a97edf" Oct 30 13:30:03.390600 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.390572 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw"] Oct 30 13:30:03.411509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.410809 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:03.411509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.410917 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:03.411509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:03.410964 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:04.224557 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:04.224482 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" event={"ID":"998f2f9e-5b70-42d1-a7ed-66de6f1c9146","Type":"ContainerStarted","Data":"a82f59a5a440afd0d36390d65e12e25d7303381f938f637f2675948b6ae201ce"} Oct 30 13:30:04.226595 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:04.226516 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" event={"ID":"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231","Type":"ContainerStarted","Data":"1f3037052e5d6bba927c98efbd6ee2226509efac2634621a8ad0f4c965d9bb20"} Oct 30 13:30:04.226595 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:04.226568 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" event={"ID":"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231","Type":"ContainerStarted","Data":"9ae0c6af16c43df717fc211d50d68686c764a332dfd288496308e412115743cf"} Oct 30 13:30:04.229172 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:04.229145 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" event={"ID":"6b7ea0eb-4b45-4168-9df5-2a6591edd74d","Type":"ContainerStarted","Data":"6cf197585df7813d6cbc2eda6f113d8c0a71fbf57d7d4b7e0b4c930063a3b45e"} Oct 30 13:30:04.231883 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:04.231410 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" event={"ID":"6b3d7096-56e6-4491-a5f1-737dcb106b1d","Type":"ContainerStarted","Data":"c6f721f10084bff75b76dcae75d8f9787678a380505889423248d37b5e071d65"} Oct 30 13:30:04.237507 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:04.234219 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" event={"ID":"6dd57d65-b68c-48f4-b173-827652241344","Type":"ContainerStarted","Data":"0676b8d74d1610fc328f36ed57e804ebe55fb20ca5f4bf10699448c769984293"} Oct 30 13:30:04.237507 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:04.236630 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" event={"ID":"38b1ca04-4bf9-48c5-abe8-644a12a66127","Type":"ContainerStarted","Data":"5428fd390237a2db7b42b5e088a0c66d413364c62d01f999250c58343ecda959"} Oct 30 13:30:04.239850 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:04.239816 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" event={"ID":"d7779049-af76-4965-8838-00e5c1eda794","Type":"ContainerStarted","Data":"0e3e61e691b6d7250cfe03ae9e0f3787feef247dba1be162fd879f3a6b038899"} Oct 30 13:30:04.514154 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:04.514046 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" podStartSLOduration=2.5140278289999998 podStartE2EDuration="2.514027829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.511714008 +0000 UTC m=+2261.143901171" watchObservedRunningTime="2026-10-30 13:30:04.514027829 +0000 UTC m=+2261.146215065" Oct 30 13:30:05.000880 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.000846 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:05.007672 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.007639 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:05.248821 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.248787 7010 generic.go:356] "Generic (PLEG): container finished" podID="6b3d7096-56e6-4491-a5f1-737dcb106b1d" containerID="c6f721f10084bff75b76dcae75d8f9787678a380505889423248d37b5e071d65" exitCode=60 Oct 30 13:30:05.249312 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.248877 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" event={"ID":"6b3d7096-56e6-4491-a5f1-737dcb106b1d","Type":"ContainerDied","Data":"c6f721f10084bff75b76dcae75d8f9787678a380505889423248d37b5e071d65"} Oct 30 13:30:05.249312 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.248948 7010 scope.go:122] "RemoveContainer" containerID="ff2fc0f54a393712d2bd57d797bc0635e958a16d0b8743c52be01b06b2044d4d" Oct 30 13:30:05.249312 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.249196 7010 scope.go:122] "RemoveContainer" containerID="c6f721f10084bff75b76dcae75d8f9787678a380505889423248d37b5e071d65" Oct 30 13:30:05.249627 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:05.249601 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8_odh-ai-gateway-infra(6b3d7096-56e6-4491-a5f1-737dcb106b1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" podUID="6b3d7096-56e6-4491-a5f1-737dcb106b1d" Oct 30 13:30:05.251128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.251068 7010 generic.go:356] "Generic (PLEG): container finished" podID="6dd57d65-b68c-48f4-b173-827652241344" containerID="0676b8d74d1610fc328f36ed57e804ebe55fb20ca5f4bf10699448c769984293" exitCode=60 Oct 30 13:30:05.251232 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.251154 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" event={"ID":"6dd57d65-b68c-48f4-b173-827652241344","Type":"ContainerDied","Data":"0676b8d74d1610fc328f36ed57e804ebe55fb20ca5f4bf10699448c769984293"} Oct 30 13:30:05.251462 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.251443 7010 scope.go:122] "RemoveContainer" containerID="0676b8d74d1610fc328f36ed57e804ebe55fb20ca5f4bf10699448c769984293" Oct 30 13:30:05.251847 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:05.251820 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9_odh-ai-gateway-infra(6dd57d65-b68c-48f4-b173-827652241344)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" podUID="6dd57d65-b68c-48f4-b173-827652241344" Oct 30 13:30:05.253228 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.253204 7010 generic.go:356] "Generic (PLEG): container finished" podID="38b1ca04-4bf9-48c5-abe8-644a12a66127" containerID="5428fd390237a2db7b42b5e088a0c66d413364c62d01f999250c58343ecda959" exitCode=60 Oct 30 13:30:05.253337 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.253313 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" event={"ID":"38b1ca04-4bf9-48c5-abe8-644a12a66127","Type":"ContainerDied","Data":"5428fd390237a2db7b42b5e088a0c66d413364c62d01f999250c58343ecda959"} Oct 30 13:30:05.253611 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.253593 7010 scope.go:122] "RemoveContainer" containerID="5428fd390237a2db7b42b5e088a0c66d413364c62d01f999250c58343ecda959" Oct 30 13:30:05.253945 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:05.253921 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh_odh-ai-gateway-infra(38b1ca04-4bf9-48c5-abe8-644a12a66127)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" podUID="38b1ca04-4bf9-48c5-abe8-644a12a66127" Oct 30 13:30:05.255747 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.255725 7010 generic.go:356] "Generic (PLEG): container finished" podID="d7779049-af76-4965-8838-00e5c1eda794" containerID="0e3e61e691b6d7250cfe03ae9e0f3787feef247dba1be162fd879f3a6b038899" exitCode=60 Oct 30 13:30:05.255833 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.255817 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" event={"ID":"d7779049-af76-4965-8838-00e5c1eda794","Type":"ContainerDied","Data":"0e3e61e691b6d7250cfe03ae9e0f3787feef247dba1be162fd879f3a6b038899"} Oct 30 13:30:05.256062 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.256043 7010 scope.go:122] "RemoveContainer" containerID="0e3e61e691b6d7250cfe03ae9e0f3787feef247dba1be162fd879f3a6b038899" Oct 30 13:30:05.256358 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:05.256335 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb_odh-ai-gateway-infra(d7779049-af76-4965-8838-00e5c1eda794)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" podUID="d7779049-af76-4965-8838-00e5c1eda794" Oct 30 13:30:05.258602 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.258577 7010 generic.go:356] "Generic (PLEG): container finished" podID="998f2f9e-5b70-42d1-a7ed-66de6f1c9146" containerID="a82f59a5a440afd0d36390d65e12e25d7303381f938f637f2675948b6ae201ce" exitCode=60 Oct 30 13:30:05.258714 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.258654 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" event={"ID":"998f2f9e-5b70-42d1-a7ed-66de6f1c9146","Type":"ContainerDied","Data":"a82f59a5a440afd0d36390d65e12e25d7303381f938f637f2675948b6ae201ce"} Oct 30 13:30:05.258919 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.258899 7010 scope.go:122] "RemoveContainer" containerID="a82f59a5a440afd0d36390d65e12e25d7303381f938f637f2675948b6ae201ce" Oct 30 13:30:05.259236 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:05.259213 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk_odh-ai-gateway-infra(998f2f9e-5b70-42d1-a7ed-66de6f1c9146)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" podUID="998f2f9e-5b70-42d1-a7ed-66de6f1c9146" Oct 30 13:30:05.261160 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.261130 7010 generic.go:356] "Generic (PLEG): container finished" podID="6b7ea0eb-4b45-4168-9df5-2a6591edd74d" containerID="6cf197585df7813d6cbc2eda6f113d8c0a71fbf57d7d4b7e0b4c930063a3b45e" exitCode=60 Oct 30 13:30:05.261273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.261213 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" event={"ID":"6b7ea0eb-4b45-4168-9df5-2a6591edd74d","Type":"ContainerDied","Data":"6cf197585df7813d6cbc2eda6f113d8c0a71fbf57d7d4b7e0b4c930063a3b45e"} Oct 30 13:30:05.261477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.261447 7010 scope.go:122] "RemoveContainer" containerID="6cf197585df7813d6cbc2eda6f113d8c0a71fbf57d7d4b7e0b4c930063a3b45e" Oct 30 13:30:05.261642 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.261625 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:05.261825 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:05.261799 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j_odh-ai-gateway-infra(6b7ea0eb-4b45-4168-9df5-2a6591edd74d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" podUID="6b7ea0eb-4b45-4168-9df5-2a6591edd74d" Oct 30 13:30:05.262949 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.262928 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:30:05.264088 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.264069 7010 scope.go:122] "RemoveContainer" containerID="fbf132faae50360f33ac4fb21a66bce32b9cedef2a0a5a11650fab7530806a14" Oct 30 13:30:05.280619 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.280592 7010 scope.go:122] "RemoveContainer" containerID="0e22329b87b19e0e7bc801e55791965a70f4b5577e5d337b77ad648cc401142e" Oct 30 13:30:05.296071 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.296038 7010 scope.go:122] "RemoveContainer" containerID="8b58f87a68698bedc7fbd200cdbdc188ebcab863314448f5b98aa168d3a8c010" Oct 30 13:30:05.311353 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.311330 7010 scope.go:122] "RemoveContainer" containerID="6fff11195a0a41fb7d73e4b5c655b6d7674179dc28e93dd868f86421ec83221e" Oct 30 13:30:05.327215 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:05.327196 7010 scope.go:122] "RemoveContainer" containerID="d4788db39907165e3187436901ad7f8e76c091230462fefedd5830a184a97edf" Oct 30 13:30:06.271533 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:06.271441 7010 scope.go:122] "RemoveContainer" containerID="c6f721f10084bff75b76dcae75d8f9787678a380505889423248d37b5e071d65" Oct 30 13:30:06.271974 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:06.271844 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8_odh-ai-gateway-infra(6b3d7096-56e6-4491-a5f1-737dcb106b1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" podUID="6b3d7096-56e6-4491-a5f1-737dcb106b1d" Oct 30 13:30:06.273231 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:06.273212 7010 scope.go:122] "RemoveContainer" containerID="0676b8d74d1610fc328f36ed57e804ebe55fb20ca5f4bf10699448c769984293" Oct 30 13:30:06.273715 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:06.273685 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9_odh-ai-gateway-infra(6dd57d65-b68c-48f4-b173-827652241344)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" podUID="6dd57d65-b68c-48f4-b173-827652241344" Oct 30 13:30:06.279143 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:06.279119 7010 scope.go:122] "RemoveContainer" containerID="5428fd390237a2db7b42b5e088a0c66d413364c62d01f999250c58343ecda959" Oct 30 13:30:06.279453 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:06.279423 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh_odh-ai-gateway-infra(38b1ca04-4bf9-48c5-abe8-644a12a66127)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" podUID="38b1ca04-4bf9-48c5-abe8-644a12a66127" Oct 30 13:30:06.280977 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:06.280957 7010 scope.go:122] "RemoveContainer" containerID="0e3e61e691b6d7250cfe03ae9e0f3787feef247dba1be162fd879f3a6b038899" Oct 30 13:30:06.281234 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:06.281211 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb_odh-ai-gateway-infra(d7779049-af76-4965-8838-00e5c1eda794)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" podUID="d7779049-af76-4965-8838-00e5c1eda794" Oct 30 13:30:06.282501 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:06.282473 7010 scope.go:122] "RemoveContainer" containerID="a82f59a5a440afd0d36390d65e12e25d7303381f938f637f2675948b6ae201ce" Oct 30 13:30:06.282751 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:06.282727 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk_odh-ai-gateway-infra(998f2f9e-5b70-42d1-a7ed-66de6f1c9146)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" podUID="998f2f9e-5b70-42d1-a7ed-66de6f1c9146" Oct 30 13:30:06.284125 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:06.284099 7010 scope.go:122] "RemoveContainer" containerID="6cf197585df7813d6cbc2eda6f113d8c0a71fbf57d7d4b7e0b4c930063a3b45e" Oct 30 13:30:06.284345 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:06.284324 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j_odh-ai-gateway-infra(6b7ea0eb-4b45-4168-9df5-2a6591edd74d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" podUID="6b7ea0eb-4b45-4168-9df5-2a6591edd74d" Oct 30 13:30:11.564010 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:11.563963 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt"] Oct 30 13:30:11.564562 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:11.564174 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" podUID="433f2adf-fc0f-4e5c-9bf3-056ab08e7891" containerName="istio-proxy" containerID="cri-o://13a248e48ec3e6c460807a5bb88b48eade8c9ba0e00e238ffadbff5edc6ed63d" gracePeriod=30 Oct 30 13:30:12.318714 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:12.318669 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm"] Oct 30 13:30:12.319023 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:12.318993 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" podUID="19e516fd-dda7-4d15-8562-6050961291aa" containerName="istio-proxy" containerID="cri-o://b4840f0accf48d6bb59b48b4d46fab74e7671b5f376feb3994c7ce27b7e988ab" gracePeriod=30 Oct 30 13:30:12.423511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:12.417894 7010 patch_prober.go:29] interesting pod/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:30:12.423511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:12.417978 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" podUID="19e516fd-dda7-4d15-8562-6050961291aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:30:15.481419 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:15.481383 7010 scope.go:122] "RemoveContainer" containerID="a82f59a5a440afd0d36390d65e12e25d7303381f938f637f2675948b6ae201ce" Oct 30 13:30:15.481882 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:15.481453 7010 scope.go:122] "RemoveContainer" containerID="0e3e61e691b6d7250cfe03ae9e0f3787feef247dba1be162fd879f3a6b038899" Oct 30 13:30:15.639954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:15.639918 7010 patch_prober.go:29] interesting pod/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:30:15.640130 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:15.639979 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" podUID="433f2adf-fc0f-4e5c-9bf3-056ab08e7891" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:30:16.401152 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:16.401116 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" event={"ID":"d7779049-af76-4965-8838-00e5c1eda794","Type":"ContainerStarted","Data":"ecb6ac0f3c2d5e1ae9e254084b8aad6f838d63c383ef52a7b55b80590c4a9ddd"} Oct 30 13:30:16.403189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:16.403164 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" event={"ID":"998f2f9e-5b70-42d1-a7ed-66de6f1c9146","Type":"ContainerStarted","Data":"8c0d165b49d264ac06a4f774f654fce03303e101e4fb898f7d9baa92b402f66e"} Oct 30 13:30:16.403578 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:16.403562 7010 scope.go:122] "RemoveContainer" containerID="8c0d165b49d264ac06a4f774f654fce03303e101e4fb898f7d9baa92b402f66e" Oct 30 13:30:16.403866 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:16.403846 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk_odh-ai-gateway-infra(998f2f9e-5b70-42d1-a7ed-66de6f1c9146)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" podUID="998f2f9e-5b70-42d1-a7ed-66de6f1c9146" Oct 30 13:30:16.481680 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:16.481655 7010 scope.go:122] "RemoveContainer" containerID="0676b8d74d1610fc328f36ed57e804ebe55fb20ca5f4bf10699448c769984293" Oct 30 13:30:16.482006 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:16.481781 7010 scope.go:122] "RemoveContainer" containerID="6cf197585df7813d6cbc2eda6f113d8c0a71fbf57d7d4b7e0b4c930063a3b45e" Oct 30 13:30:16.919446 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:16.919424 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:30:17.039616 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.039520 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-workload-socket\") pod \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " Oct 30 13:30:17.039616 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.039593 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-kube-api-access-pbd94\" (UniqueName: \"kubernetes.io/projected/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-kube-api-access-pbd94\") pod \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " Oct 30 13:30:17.039849 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.039715 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-podinfo\") pod \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " Oct 30 13:30:17.039849 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.039771 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-envoy\") pod \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " Oct 30 13:30:17.039849 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.039787 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-workload-socket" pod "433f2adf-fc0f-4e5c-9bf3-056ab08e7891" (UID: "433f2adf-fc0f-4e5c-9bf3-056ab08e7891"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.040016 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.039860 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-workload-certs\") pod \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " Oct 30 13:30:17.040016 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.039898 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-data\") pod \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " Oct 30 13:30:17.040016 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.039982 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-credential-socket\") pod \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " Oct 30 13:30:17.040208 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.040066 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istiod-ca-cert\") pod \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " Oct 30 13:30:17.040208 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.040164 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-token\" (UniqueName: \"kubernetes.io/projected/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-token\") pod \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\" (UID: \"433f2adf-fc0f-4e5c-9bf3-056ab08e7891\") " Oct 30 13:30:17.040208 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.040184 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-workload-certs" pod "433f2adf-fc0f-4e5c-9bf3-056ab08e7891" (UID: "433f2adf-fc0f-4e5c-9bf3-056ab08e7891"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.040364 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.040208 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-data" pod "433f2adf-fc0f-4e5c-9bf3-056ab08e7891" (UID: "433f2adf-fc0f-4e5c-9bf3-056ab08e7891"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.040663 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.040529 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istiod-ca-cert" pod "433f2adf-fc0f-4e5c-9bf3-056ab08e7891" (UID: "433f2adf-fc0f-4e5c-9bf3-056ab08e7891"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:17.040663 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.040616 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-credential-socket" pod "433f2adf-fc0f-4e5c-9bf3-056ab08e7891" (UID: "433f2adf-fc0f-4e5c-9bf3-056ab08e7891"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.040829 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.040805 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.040829 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.040820 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.040829 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.040828 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.041012 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.040837 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.041012 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.040846 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.042179 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.042152 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-podinfo" pod "433f2adf-fc0f-4e5c-9bf3-056ab08e7891" (UID: "433f2adf-fc0f-4e5c-9bf3-056ab08e7891"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:30:17.042291 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.042204 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-envoy" pod "433f2adf-fc0f-4e5c-9bf3-056ab08e7891" (UID: "433f2adf-fc0f-4e5c-9bf3-056ab08e7891"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.042370 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.042290 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-kube-api-access-pbd94" pod "433f2adf-fc0f-4e5c-9bf3-056ab08e7891" (UID: "433f2adf-fc0f-4e5c-9bf3-056ab08e7891"). InnerVolumeSpecName "kube-api-access-pbd94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:17.042708 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.042691 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-token" pod "433f2adf-fc0f-4e5c-9bf3-056ab08e7891" (UID: "433f2adf-fc0f-4e5c-9bf3-056ab08e7891"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:17.141954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.141913 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbd94\" (UniqueName: \"kubernetes.io/projected/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-kube-api-access-pbd94\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.141954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.141944 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.141954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.141954 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.141954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.141963 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433f2adf-fc0f-4e5c-9bf3-056ab08e7891-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.415247 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.415205 7010 generic.go:356] "Generic (PLEG): container finished" podID="433f2adf-fc0f-4e5c-9bf3-056ab08e7891" containerID="13a248e48ec3e6c460807a5bb88b48eade8c9ba0e00e238ffadbff5edc6ed63d" exitCode=0 Oct 30 13:30:17.415455 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.415279 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" Oct 30 13:30:17.415455 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.415290 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" event={"ID":"433f2adf-fc0f-4e5c-9bf3-056ab08e7891","Type":"ContainerDied","Data":"13a248e48ec3e6c460807a5bb88b48eade8c9ba0e00e238ffadbff5edc6ed63d"} Oct 30 13:30:17.415455 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.415355 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt" event={"ID":"433f2adf-fc0f-4e5c-9bf3-056ab08e7891","Type":"ContainerDied","Data":"9fe3ad92adc52e12b222bae8fa55c947a50bc0649e4ee208cf4543794cc959d5"} Oct 30 13:30:17.415455 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.415387 7010 scope.go:122] "RemoveContainer" containerID="13a248e48ec3e6c460807a5bb88b48eade8c9ba0e00e238ffadbff5edc6ed63d" Oct 30 13:30:17.417869 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.417843 7010 generic.go:356] "Generic (PLEG): container finished" podID="998f2f9e-5b70-42d1-a7ed-66de6f1c9146" containerID="8c0d165b49d264ac06a4f774f654fce03303e101e4fb898f7d9baa92b402f66e" exitCode=60 Oct 30 13:30:17.417985 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.417884 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" event={"ID":"998f2f9e-5b70-42d1-a7ed-66de6f1c9146","Type":"ContainerDied","Data":"8c0d165b49d264ac06a4f774f654fce03303e101e4fb898f7d9baa92b402f66e"} Oct 30 13:30:17.418374 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.418354 7010 scope.go:122] "RemoveContainer" containerID="8c0d165b49d264ac06a4f774f654fce03303e101e4fb898f7d9baa92b402f66e" Oct 30 13:30:17.418776 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:17.418757 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk_odh-ai-gateway-infra(998f2f9e-5b70-42d1-a7ed-66de6f1c9146)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" podUID="998f2f9e-5b70-42d1-a7ed-66de6f1c9146" Oct 30 13:30:17.420399 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.420378 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" event={"ID":"6b7ea0eb-4b45-4168-9df5-2a6591edd74d","Type":"ContainerStarted","Data":"68b20706608f36774bb40def23cc6690e3f66260f7354d6a2c07ad7cc523246d"} Oct 30 13:30:17.422528 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.422507 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" event={"ID":"6dd57d65-b68c-48f4-b173-827652241344","Type":"ContainerStarted","Data":"62f3acd4df38d3d0e22b253f5d0de77348d98dce923726a4283102554da9405e"} Oct 30 13:30:17.425984 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.425952 7010 generic.go:356] "Generic (PLEG): container finished" podID="d7779049-af76-4965-8838-00e5c1eda794" containerID="ecb6ac0f3c2d5e1ae9e254084b8aad6f838d63c383ef52a7b55b80590c4a9ddd" exitCode=60 Oct 30 13:30:17.426109 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.426001 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" event={"ID":"d7779049-af76-4965-8838-00e5c1eda794","Type":"ContainerDied","Data":"ecb6ac0f3c2d5e1ae9e254084b8aad6f838d63c383ef52a7b55b80590c4a9ddd"} Oct 30 13:30:17.426559 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.426537 7010 scope.go:122] "RemoveContainer" containerID="ecb6ac0f3c2d5e1ae9e254084b8aad6f838d63c383ef52a7b55b80590c4a9ddd" Oct 30 13:30:17.426906 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:17.426885 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb_odh-ai-gateway-infra(d7779049-af76-4965-8838-00e5c1eda794)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" podUID="d7779049-af76-4965-8838-00e5c1eda794" Oct 30 13:30:17.433158 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.433137 7010 scope.go:122] "RemoveContainer" containerID="13a248e48ec3e6c460807a5bb88b48eade8c9ba0e00e238ffadbff5edc6ed63d" Oct 30 13:30:17.433464 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:17.433440 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a248e48ec3e6c460807a5bb88b48eade8c9ba0e00e238ffadbff5edc6ed63d\": container with ID starting with 13a248e48ec3e6c460807a5bb88b48eade8c9ba0e00e238ffadbff5edc6ed63d not found: ID does not exist" containerID="13a248e48ec3e6c460807a5bb88b48eade8c9ba0e00e238ffadbff5edc6ed63d" Oct 30 13:30:17.433582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.433483 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a248e48ec3e6c460807a5bb88b48eade8c9ba0e00e238ffadbff5edc6ed63d"} err="failed to get container status \"13a248e48ec3e6c460807a5bb88b48eade8c9ba0e00e238ffadbff5edc6ed63d\": rpc error: code = NotFound desc = could not find container \"13a248e48ec3e6c460807a5bb88b48eade8c9ba0e00e238ffadbff5edc6ed63d\": container with ID starting with 13a248e48ec3e6c460807a5bb88b48eade8c9ba0e00e238ffadbff5edc6ed63d not found: ID does not exist" Oct 30 13:30:17.433582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.433518 7010 scope.go:122] "RemoveContainer" containerID="a82f59a5a440afd0d36390d65e12e25d7303381f938f637f2675948b6ae201ce" Oct 30 13:30:17.447191 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.447168 7010 scope.go:122] "RemoveContainer" containerID="0e3e61e691b6d7250cfe03ae9e0f3787feef247dba1be162fd879f3a6b038899" Oct 30 13:30:17.457250 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.457204 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb"] Oct 30 13:30:17.481262 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.481226 7010 scope.go:122] "RemoveContainer" containerID="c6f721f10084bff75b76dcae75d8f9787678a380505889423248d37b5e071d65" Oct 30 13:30:17.502468 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.502428 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk"] Oct 30 13:30:17.546795 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.545720 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt"] Oct 30 13:30:17.549251 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.549219 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52a649-openshift-default-6cdc558db9-t27zt"] Oct 30 13:30:17.787389 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.787359 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:30:17.953326 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.953227 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-credential-socket\") pod \"19e516fd-dda7-4d15-8562-6050961291aa\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " Oct 30 13:30:17.953326 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.953294 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-istio-data\") pod \"19e516fd-dda7-4d15-8562-6050961291aa\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " Oct 30 13:30:17.953639 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.953348 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19e516fd-dda7-4d15-8562-6050961291aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e516fd-dda7-4d15-8562-6050961291aa-istio-podinfo\") pod \"19e516fd-dda7-4d15-8562-6050961291aa\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " Oct 30 13:30:17.953639 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.953437 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-istio-envoy\") pod \"19e516fd-dda7-4d15-8562-6050961291aa\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " Oct 30 13:30:17.953639 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.953482 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-workload-certs\") pod \"19e516fd-dda7-4d15-8562-6050961291aa\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " Oct 30 13:30:17.953639 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.953624 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-credential-socket" pod "19e516fd-dda7-4d15-8562-6050961291aa" (UID: "19e516fd-dda7-4d15-8562-6050961291aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.953871 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.953770 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e516fd-dda7-4d15-8562-6050961291aa-istio-token\" (UniqueName: \"kubernetes.io/projected/19e516fd-dda7-4d15-8562-6050961291aa-istio-token\") pod \"19e516fd-dda7-4d15-8562-6050961291aa\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " Oct 30 13:30:17.953871 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.953791 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e516fd-dda7-4d15-8562-6050961291aa-kube-api-access-g9w8z\" (UniqueName: \"kubernetes.io/projected/19e516fd-dda7-4d15-8562-6050961291aa-kube-api-access-g9w8z\") pod \"19e516fd-dda7-4d15-8562-6050961291aa\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " Oct 30 13:30:17.953871 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.953835 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19e516fd-dda7-4d15-8562-6050961291aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e516fd-dda7-4d15-8562-6050961291aa-istiod-ca-cert\") pod \"19e516fd-dda7-4d15-8562-6050961291aa\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " Oct 30 13:30:17.953871 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.953861 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-workload-socket\") pod \"19e516fd-dda7-4d15-8562-6050961291aa\" (UID: \"19e516fd-dda7-4d15-8562-6050961291aa\") " Oct 30 13:30:17.954084 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.953900 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-workload-certs" pod "19e516fd-dda7-4d15-8562-6050961291aa" (UID: "19e516fd-dda7-4d15-8562-6050961291aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.954084 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.954050 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-istio-data" pod "19e516fd-dda7-4d15-8562-6050961291aa" (UID: "19e516fd-dda7-4d15-8562-6050961291aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.954285 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.954254 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-workload-socket" pod "19e516fd-dda7-4d15-8562-6050961291aa" (UID: "19e516fd-dda7-4d15-8562-6050961291aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.954347 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.954303 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e516fd-dda7-4d15-8562-6050961291aa-istiod-ca-cert" pod "19e516fd-dda7-4d15-8562-6050961291aa" (UID: "19e516fd-dda7-4d15-8562-6050961291aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:17.954612 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.954586 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19e516fd-dda7-4d15-8562-6050961291aa-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.954704 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.954615 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.954704 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.954630 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.954704 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.954648 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.954704 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.954660 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:17.956160 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.956132 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e516fd-dda7-4d15-8562-6050961291aa-istio-token" pod "19e516fd-dda7-4d15-8562-6050961291aa" (UID: "19e516fd-dda7-4d15-8562-6050961291aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:17.956160 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.956150 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-istio-envoy" pod "19e516fd-dda7-4d15-8562-6050961291aa" (UID: "19e516fd-dda7-4d15-8562-6050961291aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:30:17.956292 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.956138 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19e516fd-dda7-4d15-8562-6050961291aa-istio-podinfo" pod "19e516fd-dda7-4d15-8562-6050961291aa" (UID: "19e516fd-dda7-4d15-8562-6050961291aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:30:17.956292 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:17.956219 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e516fd-dda7-4d15-8562-6050961291aa-kube-api-access-g9w8z" pod "19e516fd-dda7-4d15-8562-6050961291aa" (UID: "19e516fd-dda7-4d15-8562-6050961291aa"). InnerVolumeSpecName "kube-api-access-g9w8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:18.056103 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.056068 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19e516fd-dda7-4d15-8562-6050961291aa-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:18.056103 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.056101 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9w8z\" (UniqueName: \"kubernetes.io/projected/19e516fd-dda7-4d15-8562-6050961291aa-kube-api-access-g9w8z\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:18.056103 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.056112 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19e516fd-dda7-4d15-8562-6050961291aa-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:18.056349 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.056122 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19e516fd-dda7-4d15-8562-6050961291aa-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:18.436256 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.436223 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" event={"ID":"6b3d7096-56e6-4491-a5f1-737dcb106b1d","Type":"ContainerStarted","Data":"16f74266436adf2e383b02d2662411d8d6f96be22d4e2140ad900a9374e1e1c5"} Oct 30 13:30:18.438244 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.438219 7010 generic.go:356] "Generic (PLEG): container finished" podID="6dd57d65-b68c-48f4-b173-827652241344" containerID="62f3acd4df38d3d0e22b253f5d0de77348d98dce923726a4283102554da9405e" exitCode=60 Oct 30 13:30:18.438400 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.438296 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" event={"ID":"6dd57d65-b68c-48f4-b173-827652241344","Type":"ContainerDied","Data":"62f3acd4df38d3d0e22b253f5d0de77348d98dce923726a4283102554da9405e"} Oct 30 13:30:18.438400 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.438384 7010 scope.go:122] "RemoveContainer" containerID="0676b8d74d1610fc328f36ed57e804ebe55fb20ca5f4bf10699448c769984293" Oct 30 13:30:18.438825 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.438807 7010 scope.go:122] "RemoveContainer" containerID="62f3acd4df38d3d0e22b253f5d0de77348d98dce923726a4283102554da9405e" Oct 30 13:30:18.439212 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:18.439179 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9_odh-ai-gateway-infra(6dd57d65-b68c-48f4-b173-827652241344)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" podUID="6dd57d65-b68c-48f4-b173-827652241344" Oct 30 13:30:18.439878 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.439856 7010 generic.go:356] "Generic (PLEG): container finished" podID="19e516fd-dda7-4d15-8562-6050961291aa" containerID="b4840f0accf48d6bb59b48b4d46fab74e7671b5f376feb3994c7ce27b7e988ab" exitCode=0 Oct 30 13:30:18.439955 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.439892 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" Oct 30 13:30:18.439955 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.439940 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" event={"ID":"19e516fd-dda7-4d15-8562-6050961291aa","Type":"ContainerDied","Data":"b4840f0accf48d6bb59b48b4d46fab74e7671b5f376feb3994c7ce27b7e988ab"} Oct 30 13:30:18.440074 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.439989 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm" event={"ID":"19e516fd-dda7-4d15-8562-6050961291aa","Type":"ContainerDied","Data":"7c9e15e267b19b9cb311ea207325fdd78c0a4172d1a38876cd013393f030a9af"} Oct 30 13:30:18.447230 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.447202 7010 generic.go:356] "Generic (PLEG): container finished" podID="6b7ea0eb-4b45-4168-9df5-2a6591edd74d" containerID="68b20706608f36774bb40def23cc6690e3f66260f7354d6a2c07ad7cc523246d" exitCode=60 Oct 30 13:30:18.447359 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.447230 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" event={"ID":"6b7ea0eb-4b45-4168-9df5-2a6591edd74d","Type":"ContainerDied","Data":"68b20706608f36774bb40def23cc6690e3f66260f7354d6a2c07ad7cc523246d"} Oct 30 13:30:18.447689 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.447669 7010 scope.go:122] "RemoveContainer" containerID="68b20706608f36774bb40def23cc6690e3f66260f7354d6a2c07ad7cc523246d" Oct 30 13:30:18.448020 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:18.447998 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j_odh-ai-gateway-infra(6b7ea0eb-4b45-4168-9df5-2a6591edd74d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" podUID="6b7ea0eb-4b45-4168-9df5-2a6591edd74d" Oct 30 13:30:18.455143 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.455126 7010 scope.go:122] "RemoveContainer" containerID="b4840f0accf48d6bb59b48b4d46fab74e7671b5f376feb3994c7ce27b7e988ab" Oct 30 13:30:18.465585 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.465553 7010 scope.go:122] "RemoveContainer" containerID="b4840f0accf48d6bb59b48b4d46fab74e7671b5f376feb3994c7ce27b7e988ab" Oct 30 13:30:18.465928 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:18.465906 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4840f0accf48d6bb59b48b4d46fab74e7671b5f376feb3994c7ce27b7e988ab\": container with ID starting with b4840f0accf48d6bb59b48b4d46fab74e7671b5f376feb3994c7ce27b7e988ab not found: ID does not exist" containerID="b4840f0accf48d6bb59b48b4d46fab74e7671b5f376feb3994c7ce27b7e988ab" Oct 30 13:30:18.466069 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.465940 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4840f0accf48d6bb59b48b4d46fab74e7671b5f376feb3994c7ce27b7e988ab"} err="failed to get container status \"b4840f0accf48d6bb59b48b4d46fab74e7671b5f376feb3994c7ce27b7e988ab\": rpc error: code = NotFound desc = could not find container \"b4840f0accf48d6bb59b48b4d46fab74e7671b5f376feb3994c7ce27b7e988ab\": container with ID starting with b4840f0accf48d6bb59b48b4d46fab74e7671b5f376feb3994c7ce27b7e988ab not found: ID does not exist" Oct 30 13:30:18.466069 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.465967 7010 scope.go:122] "RemoveContainer" containerID="6cf197585df7813d6cbc2eda6f113d8c0a71fbf57d7d4b7e0b4c930063a3b45e" Oct 30 13:30:18.482148 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.481749 7010 scope.go:122] "RemoveContainer" containerID="5428fd390237a2db7b42b5e088a0c66d413364c62d01f999250c58343ecda959" Oct 30 13:30:18.636262 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.636227 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j"] Oct 30 13:30:18.660216 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.660186 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm"] Oct 30 13:30:18.671088 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.671054 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-376c89-openshift-default-86bbc44557-k6djm"] Oct 30 13:30:18.680595 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.680565 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9"] Oct 30 13:30:18.724773 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.724736 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" Oct 30 13:30:18.727093 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.727067 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" Oct 30 13:30:18.867578 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.867549 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998f2f9e-5b70-42d1-a7ed-66de6f1c9146-kube-api-access-hrmtx\" (UniqueName: \"kubernetes.io/projected/998f2f9e-5b70-42d1-a7ed-66de6f1c9146-kube-api-access-hrmtx\") pod \"998f2f9e-5b70-42d1-a7ed-66de6f1c9146\" (UID: \"998f2f9e-5b70-42d1-a7ed-66de6f1c9146\") " Oct 30 13:30:18.867790 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.867597 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7779049-af76-4965-8838-00e5c1eda794-kube-api-access-vh7xc\" (UniqueName: \"kubernetes.io/projected/d7779049-af76-4965-8838-00e5c1eda794-kube-api-access-vh7xc\") pod \"d7779049-af76-4965-8838-00e5c1eda794\" (UID: \"d7779049-af76-4965-8838-00e5c1eda794\") " Oct 30 13:30:18.867863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.867810 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998f2f9e-5b70-42d1-a7ed-66de6f1c9146-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/998f2f9e-5b70-42d1-a7ed-66de6f1c9146-maas-api-service-ca\") pod \"998f2f9e-5b70-42d1-a7ed-66de6f1c9146\" (UID: \"998f2f9e-5b70-42d1-a7ed-66de6f1c9146\") " Oct 30 13:30:18.867961 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.867938 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7779049-af76-4965-8838-00e5c1eda794-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7779049-af76-4965-8838-00e5c1eda794-maas-api-service-ca\") pod \"d7779049-af76-4965-8838-00e5c1eda794\" (UID: \"d7779049-af76-4965-8838-00e5c1eda794\") " Oct 30 13:30:18.868402 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.868374 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998f2f9e-5b70-42d1-a7ed-66de6f1c9146-maas-api-service-ca" pod "998f2f9e-5b70-42d1-a7ed-66de6f1c9146" (UID: "998f2f9e-5b70-42d1-a7ed-66de6f1c9146"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:18.868563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.868379 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7779049-af76-4965-8838-00e5c1eda794-maas-api-service-ca" pod "d7779049-af76-4965-8838-00e5c1eda794" (UID: "d7779049-af76-4965-8838-00e5c1eda794"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:18.870461 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.870438 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7779049-af76-4965-8838-00e5c1eda794-kube-api-access-vh7xc" pod "d7779049-af76-4965-8838-00e5c1eda794" (UID: "d7779049-af76-4965-8838-00e5c1eda794"). InnerVolumeSpecName "kube-api-access-vh7xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:18.874780 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.874750 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998f2f9e-5b70-42d1-a7ed-66de6f1c9146-kube-api-access-hrmtx" pod "998f2f9e-5b70-42d1-a7ed-66de6f1c9146" (UID: "998f2f9e-5b70-42d1-a7ed-66de6f1c9146"). InnerVolumeSpecName "kube-api-access-hrmtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:18.970175 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.970098 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrmtx\" (UniqueName: \"kubernetes.io/projected/998f2f9e-5b70-42d1-a7ed-66de6f1c9146-kube-api-access-hrmtx\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:18.970175 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.970125 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh7xc\" (UniqueName: \"kubernetes.io/projected/d7779049-af76-4965-8838-00e5c1eda794-kube-api-access-vh7xc\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:18.970175 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.970135 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/998f2f9e-5b70-42d1-a7ed-66de6f1c9146-maas-api-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:18.970175 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:18.970145 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7779049-af76-4965-8838-00e5c1eda794-maas-api-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:19.460466 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.460438 7010 generic.go:356] "Generic (PLEG): container finished" podID="6b3d7096-56e6-4491-a5f1-737dcb106b1d" containerID="16f74266436adf2e383b02d2662411d8d6f96be22d4e2140ad900a9374e1e1c5" exitCode=60 Oct 30 13:30:19.460592 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.460547 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" event={"ID":"6b3d7096-56e6-4491-a5f1-737dcb106b1d","Type":"ContainerDied","Data":"16f74266436adf2e383b02d2662411d8d6f96be22d4e2140ad900a9374e1e1c5"} Oct 30 13:30:19.460639 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.460605 7010 scope.go:122] "RemoveContainer" containerID="c6f721f10084bff75b76dcae75d8f9787678a380505889423248d37b5e071d65" Oct 30 13:30:19.461021 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.460990 7010 scope.go:122] "RemoveContainer" containerID="16f74266436adf2e383b02d2662411d8d6f96be22d4e2140ad900a9374e1e1c5" Oct 30 13:30:19.461386 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:19.461361 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8_odh-ai-gateway-infra(6b3d7096-56e6-4491-a5f1-737dcb106b1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" podUID="6b3d7096-56e6-4491-a5f1-737dcb106b1d" Oct 30 13:30:19.465783 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.465755 7010 generic.go:356] "Generic (PLEG): container finished" podID="38b1ca04-4bf9-48c5-abe8-644a12a66127" containerID="acb9389c5dccdd2088cc8500a03d1984aebc63ee71f42349df6b2db93dd7c529" exitCode=60 Oct 30 13:30:19.465889 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.465831 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" event={"ID":"38b1ca04-4bf9-48c5-abe8-644a12a66127","Type":"ContainerDied","Data":"acb9389c5dccdd2088cc8500a03d1984aebc63ee71f42349df6b2db93dd7c529"} Oct 30 13:30:19.466345 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.466323 7010 scope.go:122] "RemoveContainer" containerID="acb9389c5dccdd2088cc8500a03d1984aebc63ee71f42349df6b2db93dd7c529" Oct 30 13:30:19.466777 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:19.466748 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh_odh-ai-gateway-infra(38b1ca04-4bf9-48c5-abe8-644a12a66127)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" podUID="38b1ca04-4bf9-48c5-abe8-644a12a66127" Oct 30 13:30:19.467389 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.467370 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" Oct 30 13:30:19.467506 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.467433 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb" event={"ID":"d7779049-af76-4965-8838-00e5c1eda794","Type":"ContainerDied","Data":"370ddbe0e07eba22779c942ad25541a59f8b8f1530ce6023cb514e622dc369d0"} Oct 30 13:30:19.469063 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.469020 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" event={"ID":"998f2f9e-5b70-42d1-a7ed-66de6f1c9146","Type":"ContainerDied","Data":"cb8f83e23736ad671ffbe79691d39e493d595634e11f59c6977baaee71a224b5"} Oct 30 13:30:19.469153 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.469096 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk" Oct 30 13:30:19.478009 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.477927 7010 scope.go:122] "RemoveContainer" containerID="5428fd390237a2db7b42b5e088a0c66d413364c62d01f999250c58343ecda959" Oct 30 13:30:19.492168 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.492135 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19e516fd-dda7-4d15-8562-6050961291aa" path="/var/lib/kubelet/pods/19e516fd-dda7-4d15-8562-6050961291aa/volumes" Oct 30 13:30:19.508715 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.493218 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="433f2adf-fc0f-4e5c-9bf3-056ab08e7891" path="/var/lib/kubelet/pods/433f2adf-fc0f-4e5c-9bf3-056ab08e7891/volumes" Oct 30 13:30:19.508715 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.495907 7010 scope.go:122] "RemoveContainer" containerID="ecb6ac0f3c2d5e1ae9e254084b8aad6f838d63c383ef52a7b55b80590c4a9ddd" Oct 30 13:30:19.508715 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.496388 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8"] Oct 30 13:30:19.511226 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.511204 7010 scope.go:122] "RemoveContainer" containerID="8c0d165b49d264ac06a4f774f654fce03303e101e4fb898f7d9baa92b402f66e" Oct 30 13:30:19.589328 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.589293 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb"] Oct 30 13:30:19.595239 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.594758 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ff4285-29889450-tfxgb"] Oct 30 13:30:19.649115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.649038 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" Oct 30 13:30:19.686548 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.686518 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" Oct 30 13:30:19.732573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.732541 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk"] Oct 30 13:30:19.743999 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.743967 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9cbcbc-29889450-tfxfk"] Oct 30 13:30:19.785749 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.785719 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd57d65-b68c-48f4-b173-827652241344-kube-api-access-sd4r6\" (UniqueName: \"kubernetes.io/projected/6dd57d65-b68c-48f4-b173-827652241344-kube-api-access-sd4r6\") pod \"6dd57d65-b68c-48f4-b173-827652241344\" (UID: \"6dd57d65-b68c-48f4-b173-827652241344\") " Oct 30 13:30:19.785945 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.785770 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7ea0eb-4b45-4168-9df5-2a6591edd74d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7ea0eb-4b45-4168-9df5-2a6591edd74d-maas-api-service-ca\") pod \"6b7ea0eb-4b45-4168-9df5-2a6591edd74d\" (UID: \"6b7ea0eb-4b45-4168-9df5-2a6591edd74d\") " Oct 30 13:30:19.785945 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.785887 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7ea0eb-4b45-4168-9df5-2a6591edd74d-kube-api-access-hbtfb\" (UniqueName: \"kubernetes.io/projected/6b7ea0eb-4b45-4168-9df5-2a6591edd74d-kube-api-access-hbtfb\") pod \"6b7ea0eb-4b45-4168-9df5-2a6591edd74d\" (UID: \"6b7ea0eb-4b45-4168-9df5-2a6591edd74d\") " Oct 30 13:30:19.785945 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.785937 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd57d65-b68c-48f4-b173-827652241344-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd57d65-b68c-48f4-b173-827652241344-maas-api-service-ca\") pod \"6dd57d65-b68c-48f4-b173-827652241344\" (UID: \"6dd57d65-b68c-48f4-b173-827652241344\") " Oct 30 13:30:19.786274 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.786239 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7ea0eb-4b45-4168-9df5-2a6591edd74d-maas-api-service-ca" pod "6b7ea0eb-4b45-4168-9df5-2a6591edd74d" (UID: "6b7ea0eb-4b45-4168-9df5-2a6591edd74d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:19.786390 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.786319 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd57d65-b68c-48f4-b173-827652241344-maas-api-service-ca" pod "6dd57d65-b68c-48f4-b173-827652241344" (UID: "6dd57d65-b68c-48f4-b173-827652241344"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:19.786432 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.786416 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd57d65-b68c-48f4-b173-827652241344-maas-api-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:19.786432 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.786429 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7ea0eb-4b45-4168-9df5-2a6591edd74d-maas-api-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:19.787985 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.787958 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7ea0eb-4b45-4168-9df5-2a6591edd74d-kube-api-access-hbtfb" pod "6b7ea0eb-4b45-4168-9df5-2a6591edd74d" (UID: "6b7ea0eb-4b45-4168-9df5-2a6591edd74d"). InnerVolumeSpecName "kube-api-access-hbtfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:19.788037 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.788001 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd57d65-b68c-48f4-b173-827652241344-kube-api-access-sd4r6" pod "6dd57d65-b68c-48f4-b173-827652241344" (UID: "6dd57d65-b68c-48f4-b173-827652241344"). InnerVolumeSpecName "kube-api-access-sd4r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:19.888174 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.888145 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbtfb\" (UniqueName: \"kubernetes.io/projected/6b7ea0eb-4b45-4168-9df5-2a6591edd74d-kube-api-access-hbtfb\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:19.888174 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:19.888171 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd4r6\" (UniqueName: \"kubernetes.io/projected/6dd57d65-b68c-48f4-b173-827652241344-kube-api-access-sd4r6\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:20.478273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.478251 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" Oct 30 13:30:20.478405 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.478285 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9" event={"ID":"6dd57d65-b68c-48f4-b173-827652241344","Type":"ContainerDied","Data":"b62be3228dfda0f6fcb951a1afb81d2d30f1100b65758b03125f975d171882fc"} Oct 30 13:30:20.478405 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.478358 7010 scope.go:122] "RemoveContainer" containerID="62f3acd4df38d3d0e22b253f5d0de77348d98dce923726a4283102554da9405e" Oct 30 13:30:20.484433 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.484409 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" Oct 30 13:30:20.484433 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.484416 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j" event={"ID":"6b7ea0eb-4b45-4168-9df5-2a6591edd74d","Type":"ContainerDied","Data":"08be9a5a56ead25cd25970759d18ee224f92cf83acfeedb35c3d2f74bf57339d"} Oct 30 13:30:20.489767 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.489745 7010 scope.go:122] "RemoveContainer" containerID="68b20706608f36774bb40def23cc6690e3f66260f7354d6a2c07ad7cc523246d" Oct 30 13:30:20.560374 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.560345 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j"] Oct 30 13:30:20.564174 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.564147 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52a649-29889450-zt62j"] Oct 30 13:30:20.600859 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.600830 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9"] Oct 30 13:30:20.606802 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.606774 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d7e62b-29889450-9hkp9"] Oct 30 13:30:20.641049 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.641025 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" Oct 30 13:30:20.704377 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.704339 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh"] Oct 30 13:30:20.800984 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.800930 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3d7096-56e6-4491-a5f1-737dcb106b1d-kube-api-access-sjdxs\" (UniqueName: \"kubernetes.io/projected/6b3d7096-56e6-4491-a5f1-737dcb106b1d-kube-api-access-sjdxs\") pod \"6b3d7096-56e6-4491-a5f1-737dcb106b1d\" (UID: \"6b3d7096-56e6-4491-a5f1-737dcb106b1d\") " Oct 30 13:30:20.801218 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.801147 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3d7096-56e6-4491-a5f1-737dcb106b1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3d7096-56e6-4491-a5f1-737dcb106b1d-maas-api-service-ca\") pod \"6b3d7096-56e6-4491-a5f1-737dcb106b1d\" (UID: \"6b3d7096-56e6-4491-a5f1-737dcb106b1d\") " Oct 30 13:30:20.802422 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.802269 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3d7096-56e6-4491-a5f1-737dcb106b1d-maas-api-service-ca" pod "6b3d7096-56e6-4491-a5f1-737dcb106b1d" (UID: "6b3d7096-56e6-4491-a5f1-737dcb106b1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:20.812529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.805148 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3d7096-56e6-4491-a5f1-737dcb106b1d-kube-api-access-sjdxs" pod "6b3d7096-56e6-4491-a5f1-737dcb106b1d" (UID: "6b3d7096-56e6-4491-a5f1-737dcb106b1d"). InnerVolumeSpecName "kube-api-access-sjdxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:20.873885 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.873856 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" Oct 30 13:30:20.903832 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.903794 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjdxs\" (UniqueName: \"kubernetes.io/projected/6b3d7096-56e6-4491-a5f1-737dcb106b1d-kube-api-access-sjdxs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:20.903832 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:20.903831 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3d7096-56e6-4491-a5f1-737dcb106b1d-maas-api-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:21.004306 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.004264 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38b1ca04-4bf9-48c5-abe8-644a12a66127-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b1ca04-4bf9-48c5-abe8-644a12a66127-maas-api-service-ca\") pod \"38b1ca04-4bf9-48c5-abe8-644a12a66127\" (UID: \"38b1ca04-4bf9-48c5-abe8-644a12a66127\") " Oct 30 13:30:21.004526 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.004424 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b1ca04-4bf9-48c5-abe8-644a12a66127-kube-api-access-rqrj4\" (UniqueName: \"kubernetes.io/projected/38b1ca04-4bf9-48c5-abe8-644a12a66127-kube-api-access-rqrj4\") pod \"38b1ca04-4bf9-48c5-abe8-644a12a66127\" (UID: \"38b1ca04-4bf9-48c5-abe8-644a12a66127\") " Oct 30 13:30:21.004828 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.004795 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38b1ca04-4bf9-48c5-abe8-644a12a66127-maas-api-service-ca" pod "38b1ca04-4bf9-48c5-abe8-644a12a66127" (UID: "38b1ca04-4bf9-48c5-abe8-644a12a66127"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:30:21.005127 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.005095 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b1ca04-4bf9-48c5-abe8-644a12a66127-maas-api-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:21.006852 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.006823 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b1ca04-4bf9-48c5-abe8-644a12a66127-kube-api-access-rqrj4" pod "38b1ca04-4bf9-48c5-abe8-644a12a66127" (UID: "38b1ca04-4bf9-48c5-abe8-644a12a66127"). InnerVolumeSpecName "kube-api-access-rqrj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:30:21.106835 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.106750 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqrj4\" (UniqueName: \"kubernetes.io/projected/38b1ca04-4bf9-48c5-abe8-644a12a66127-kube-api-access-rqrj4\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:30:21.489905 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.489869 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b7ea0eb-4b45-4168-9df5-2a6591edd74d" path="/var/lib/kubelet/pods/6b7ea0eb-4b45-4168-9df5-2a6591edd74d/volumes" Oct 30 13:30:21.490381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.490365 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd57d65-b68c-48f4-b173-827652241344" path="/var/lib/kubelet/pods/6dd57d65-b68c-48f4-b173-827652241344/volumes" Oct 30 13:30:21.490768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.490753 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998f2f9e-5b70-42d1-a7ed-66de6f1c9146" path="/var/lib/kubelet/pods/998f2f9e-5b70-42d1-a7ed-66de6f1c9146/volumes" Oct 30 13:30:21.491458 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.491443 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7779049-af76-4965-8838-00e5c1eda794" path="/var/lib/kubelet/pods/d7779049-af76-4965-8838-00e5c1eda794/volumes" Oct 30 13:30:21.494330 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.494307 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" Oct 30 13:30:21.494459 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.494344 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh" event={"ID":"38b1ca04-4bf9-48c5-abe8-644a12a66127","Type":"ContainerDied","Data":"79540cf5918ae9f949375456274dfbeee6c8da6e03c9ae0ec311d73886c508e8"} Oct 30 13:30:21.494459 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.494398 7010 scope.go:122] "RemoveContainer" containerID="acb9389c5dccdd2088cc8500a03d1984aebc63ee71f42349df6b2db93dd7c529" Oct 30 13:30:21.497389 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.497360 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" event={"ID":"6b3d7096-56e6-4491-a5f1-737dcb106b1d","Type":"ContainerDied","Data":"8c6f737e33d70ec9272c5f21844cf09d1c9fde9ce39f7f93a8a4fcd5e214af52"} Oct 30 13:30:21.497538 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.497406 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8" Oct 30 13:30:21.506336 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.506319 7010 scope.go:122] "RemoveContainer" containerID="16f74266436adf2e383b02d2662411d8d6f96be22d4e2140ad900a9374e1e1c5" Oct 30 13:30:21.550722 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.550689 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh"] Oct 30 13:30:21.559307 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.559278 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee7d65-29889450-hh8hh"] Oct 30 13:30:21.595185 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.595146 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8"] Oct 30 13:30:21.600879 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:21.600843 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c47903-29889450-ggpn8"] Oct 30 13:30:23.489261 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:23.489227 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38b1ca04-4bf9-48c5-abe8-644a12a66127" path="/var/lib/kubelet/pods/38b1ca04-4bf9-48c5-abe8-644a12a66127/volumes" Oct 30 13:30:23.489738 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:23.489722 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3d7096-56e6-4491-a5f1-737dcb106b1d" path="/var/lib/kubelet/pods/6b3d7096-56e6-4491-a5f1-737dcb106b1d/volumes" Oct 30 13:30:25.042509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:25.040168 7010 scope.go:122] "RemoveContainer" containerID="ef437412f0788ce10ba1f38302921da48c4a016f1a808a3bd0925c9b665d2636" Oct 30 13:30:29.425061 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:29.425022 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:30:33.638399 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:33.638359 7010 generic.go:356] "Generic (PLEG): container finished" podID="415237a2-07c0-481d-bb7b-f818c83e92ce" containerID="00061988993a7660d7b7ff0f9c51ee66c0c4a28107ff661c82543a4198ffe954" exitCode=0 Oct 30 13:30:33.639004 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:33.638428 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" event={"ID":"415237a2-07c0-481d-bb7b-f818c83e92ce","Type":"ContainerDied","Data":"00061988993a7660d7b7ff0f9c51ee66c0c4a28107ff661c82543a4198ffe954"} Oct 30 13:30:33.639004 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:33.638950 7010 scope.go:122] "RemoveContainer" containerID="00061988993a7660d7b7ff0f9c51ee66c0c4a28107ff661c82543a4198ffe954" Oct 30 13:30:33.641089 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:33.641048 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:33.641205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:33.641147 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:33.641205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:33.641192 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:30:34.647738 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:34.647702 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" event={"ID":"415237a2-07c0-481d-bb7b-f818c83e92ce","Type":"ContainerStarted","Data":"1ec1e6006a2769811c0ca7d781ef431f29b11040a8508bb46bdfc7e06cb91fca"} Oct 30 13:30:34.736655 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:34.736611 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:30:34.737836 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:34.737801 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:30:34.737994 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:34.737859 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" podUID="415237a2-07c0-481d-bb7b-f818c83e92ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 13:30:35.736583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:35.736546 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:30:35.737101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:35.736614 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" podUID="415237a2-07c0-481d-bb7b-f818c83e92ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 13:30:36.284521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:36.283803 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:30:36.736727 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:36.736689 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:30:36.737133 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:36.736754 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" podUID="415237a2-07c0-481d-bb7b-f818c83e92ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 13:30:37.737106 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:37.737068 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:30:37.737553 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:37.737126 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" podUID="415237a2-07c0-481d-bb7b-f818c83e92ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 13:30:38.737042 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:38.737004 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:30:38.737237 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:38.737071 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" podUID="415237a2-07c0-481d-bb7b-f818c83e92ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 13:30:39.741833 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:39.737274 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 13:30:39.741833 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:39.737338 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" podUID="415237a2-07c0-481d-bb7b-f818c83e92ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 13:30:40.741573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:40.741540 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:30:40.742736 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:40.742710 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:30:47.171942 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.171892 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk"] Oct 30 13:30:47.172750 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172732 7010 cpu_manager.go:414] "containerMap: removing container" podUID="19e516fd-dda7-4d15-8562-6050961291aa" containerName="istio-proxy" Oct 30 13:30:47.172826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172752 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e516fd-dda7-4d15-8562-6050961291aa" containerName="istio-proxy" Oct 30 13:30:47.172826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172768 7010 cpu_manager.go:414] "containerMap: removing container" podUID="6b3d7096-56e6-4491-a5f1-737dcb106b1d" containerName="cleanup" Oct 30 13:30:47.172826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172776 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3d7096-56e6-4491-a5f1-737dcb106b1d" containerName="cleanup" Oct 30 13:30:47.172826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172797 7010 cpu_manager.go:414] "containerMap: removing container" podUID="38b1ca04-4bf9-48c5-abe8-644a12a66127" containerName="cleanup" Oct 30 13:30:47.172826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172806 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b1ca04-4bf9-48c5-abe8-644a12a66127" containerName="cleanup" Oct 30 13:30:47.172826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172823 7010 cpu_manager.go:414] "containerMap: removing container" podUID="6b7ea0eb-4b45-4168-9df5-2a6591edd74d" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172831 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7ea0eb-4b45-4168-9df5-2a6591edd74d" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172842 7010 cpu_manager.go:414] "containerMap: removing container" podUID="d7779049-af76-4965-8838-00e5c1eda794" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172850 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7779049-af76-4965-8838-00e5c1eda794" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172864 7010 cpu_manager.go:414] "containerMap: removing container" podUID="6dd57d65-b68c-48f4-b173-827652241344" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172871 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd57d65-b68c-48f4-b173-827652241344" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172879 7010 cpu_manager.go:414] "containerMap: removing container" podUID="6b3d7096-56e6-4491-a5f1-737dcb106b1d" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172886 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3d7096-56e6-4491-a5f1-737dcb106b1d" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172899 7010 cpu_manager.go:414] "containerMap: removing container" podUID="998f2f9e-5b70-42d1-a7ed-66de6f1c9146" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172905 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998f2f9e-5b70-42d1-a7ed-66de6f1c9146" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172918 7010 cpu_manager.go:414] "containerMap: removing container" podUID="6dd57d65-b68c-48f4-b173-827652241344" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172925 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd57d65-b68c-48f4-b173-827652241344" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172936 7010 cpu_manager.go:414] "containerMap: removing container" podUID="6b7ea0eb-4b45-4168-9df5-2a6591edd74d" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172943 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7ea0eb-4b45-4168-9df5-2a6591edd74d" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172953 7010 cpu_manager.go:414] "containerMap: removing container" podUID="38b1ca04-4bf9-48c5-abe8-644a12a66127" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172961 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b1ca04-4bf9-48c5-abe8-644a12a66127" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172984 7010 cpu_manager.go:414] "containerMap: removing container" podUID="6b7ea0eb-4b45-4168-9df5-2a6591edd74d" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.172990 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7ea0eb-4b45-4168-9df5-2a6591edd74d" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173002 7010 cpu_manager.go:414] "containerMap: removing container" podUID="433f2adf-fc0f-4e5c-9bf3-056ab08e7891" containerName="istio-proxy" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173008 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="433f2adf-fc0f-4e5c-9bf3-056ab08e7891" containerName="istio-proxy" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173025 7010 cpu_manager.go:414] "containerMap: removing container" podUID="998f2f9e-5b70-42d1-a7ed-66de6f1c9146" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173032 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998f2f9e-5b70-42d1-a7ed-66de6f1c9146" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173042 7010 cpu_manager.go:414] "containerMap: removing container" podUID="998f2f9e-5b70-42d1-a7ed-66de6f1c9146" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173050 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998f2f9e-5b70-42d1-a7ed-66de6f1c9146" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173065 7010 cpu_manager.go:414] "containerMap: removing container" podUID="d7779049-af76-4965-8838-00e5c1eda794" containerName="cleanup" Oct 30 13:30:47.173123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173072 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7779049-af76-4965-8838-00e5c1eda794" containerName="cleanup" Oct 30 13:30:47.174115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173201 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7ea0eb-4b45-4168-9df5-2a6591edd74d" containerName="cleanup" Oct 30 13:30:47.174115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173210 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7779049-af76-4965-8838-00e5c1eda794" containerName="cleanup" Oct 30 13:30:47.174115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173216 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="433f2adf-fc0f-4e5c-9bf3-056ab08e7891" containerName="istio-proxy" Oct 30 13:30:47.174115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173222 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="998f2f9e-5b70-42d1-a7ed-66de6f1c9146" containerName="cleanup" Oct 30 13:30:47.174115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173229 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="998f2f9e-5b70-42d1-a7ed-66de6f1c9146" containerName="cleanup" Oct 30 13:30:47.174115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173237 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd57d65-b68c-48f4-b173-827652241344" containerName="cleanup" Oct 30 13:30:47.174115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173242 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b1ca04-4bf9-48c5-abe8-644a12a66127" containerName="cleanup" Oct 30 13:30:47.174115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173247 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e516fd-dda7-4d15-8562-6050961291aa" containerName="istio-proxy" Oct 30 13:30:47.174115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173253 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3d7096-56e6-4491-a5f1-737dcb106b1d" containerName="cleanup" Oct 30 13:30:47.174115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173261 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7ea0eb-4b45-4168-9df5-2a6591edd74d" containerName="cleanup" Oct 30 13:30:47.174115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173267 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7779049-af76-4965-8838-00e5c1eda794" containerName="cleanup" Oct 30 13:30:47.174115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173273 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3d7096-56e6-4491-a5f1-737dcb106b1d" containerName="cleanup" Oct 30 13:30:47.174115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173281 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd57d65-b68c-48f4-b173-827652241344" containerName="cleanup" Oct 30 13:30:47.174115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173287 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7ea0eb-4b45-4168-9df5-2a6591edd74d" containerName="cleanup" Oct 30 13:30:47.174115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.173294 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b1ca04-4bf9-48c5-abe8-644a12a66127" containerName="cleanup" Oct 30 13:30:47.178456 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.178431 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:30:47.201795 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.201759 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-fdd4d4-serving-cert\"" Oct 30 13:30:47.231954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.231916 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk"] Oct 30 13:30:47.258031 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.257985 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz2s8\" (UniqueName: \"kubernetes.io/projected/998f2069-f1df-403c-9719-46a9a3cda3fa-kube-api-access-nz2s8\") pod \"maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk\" (UID: \"998f2069-f1df-403c-9719-46a9a3cda3fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:30:47.258250 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.258048 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/998f2069-f1df-403c-9719-46a9a3cda3fa-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk\" (UID: \"998f2069-f1df-403c-9719-46a9a3cda3fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:30:47.258250 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.258199 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/998f2069-f1df-403c-9719-46a9a3cda3fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk\" (UID: \"998f2069-f1df-403c-9719-46a9a3cda3fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:30:47.258381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.258262 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/998f2069-f1df-403c-9719-46a9a3cda3fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk\" (UID: \"998f2069-f1df-403c-9719-46a9a3cda3fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:30:47.359637 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.359604 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz2s8\" (UniqueName: \"kubernetes.io/projected/998f2069-f1df-403c-9719-46a9a3cda3fa-kube-api-access-nz2s8\") pod \"maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk\" (UID: \"998f2069-f1df-403c-9719-46a9a3cda3fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:30:47.359838 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.359646 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/998f2069-f1df-403c-9719-46a9a3cda3fa-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk\" (UID: \"998f2069-f1df-403c-9719-46a9a3cda3fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:30:47.359838 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.359721 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/998f2069-f1df-403c-9719-46a9a3cda3fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk\" (UID: \"998f2069-f1df-403c-9719-46a9a3cda3fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:30:47.359838 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.359764 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/998f2069-f1df-403c-9719-46a9a3cda3fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk\" (UID: \"998f2069-f1df-403c-9719-46a9a3cda3fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:30:47.360577 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.360543 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/998f2069-f1df-403c-9719-46a9a3cda3fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk\" (UID: \"998f2069-f1df-403c-9719-46a9a3cda3fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:30:47.362436 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.362414 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/998f2069-f1df-403c-9719-46a9a3cda3fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk\" (UID: \"998f2069-f1df-403c-9719-46a9a3cda3fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:30:47.362590 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.362559 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/998f2069-f1df-403c-9719-46a9a3cda3fa-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk\" (UID: \"998f2069-f1df-403c-9719-46a9a3cda3fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:30:47.384606 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.384562 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz2s8\" (UniqueName: \"kubernetes.io/projected/998f2069-f1df-403c-9719-46a9a3cda3fa-kube-api-access-nz2s8\") pod \"maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk\" (UID: \"998f2069-f1df-403c-9719-46a9a3cda3fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:30:47.494321 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.494227 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:30:47.863096 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:47.863058 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk"] Oct 30 13:30:47.866772 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:30:47.866730 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998f2069_f1df_403c_9719_46a9a3cda3fa.slice/crio-28ea46e187b0872a540d5931fa71c05f19be485462b3d323bbac3f796ed3639e WatchSource:0}: Error finding container 28ea46e187b0872a540d5931fa71c05f19be485462b3d323bbac3f796ed3639e: Status 404 returned error can't find the container with id 28ea46e187b0872a540d5931fa71c05f19be485462b3d323bbac3f796ed3639e Oct 30 13:30:48.012933 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.012892 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x"] Oct 30 13:30:48.013775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.013753 7010 cpu_manager.go:414] "containerMap: removing container" podUID="d7779049-af76-4965-8838-00e5c1eda794" containerName="cleanup" Oct 30 13:30:48.013775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.013775 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7779049-af76-4965-8838-00e5c1eda794" containerName="cleanup" Oct 30 13:30:48.013932 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.013818 7010 cpu_manager.go:414] "containerMap: removing container" podUID="6b3d7096-56e6-4491-a5f1-737dcb106b1d" containerName="cleanup" Oct 30 13:30:48.013932 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.013827 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3d7096-56e6-4491-a5f1-737dcb106b1d" containerName="cleanup" Oct 30 13:30:48.013932 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.013850 7010 cpu_manager.go:414] "containerMap: removing container" podUID="6dd57d65-b68c-48f4-b173-827652241344" containerName="cleanup" Oct 30 13:30:48.013932 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.013858 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd57d65-b68c-48f4-b173-827652241344" containerName="cleanup" Oct 30 13:30:48.013932 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.013876 7010 cpu_manager.go:414] "containerMap: removing container" podUID="38b1ca04-4bf9-48c5-abe8-644a12a66127" containerName="cleanup" Oct 30 13:30:48.013932 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.013884 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b1ca04-4bf9-48c5-abe8-644a12a66127" containerName="cleanup" Oct 30 13:30:48.014230 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.014027 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7779049-af76-4965-8838-00e5c1eda794" containerName="cleanup" Oct 30 13:30:48.014230 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.014040 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3d7096-56e6-4491-a5f1-737dcb106b1d" containerName="cleanup" Oct 30 13:30:48.014230 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.014057 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="998f2f9e-5b70-42d1-a7ed-66de6f1c9146" containerName="cleanup" Oct 30 13:30:48.014230 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.014067 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd57d65-b68c-48f4-b173-827652241344" containerName="cleanup" Oct 30 13:30:48.014997 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.014077 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b1ca04-4bf9-48c5-abe8-644a12a66127" containerName="cleanup" Oct 30 13:30:48.027149 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.027119 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.046206 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.045358 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c47903\"/\"kube-root-ca.crt\"" Oct 30 13:30:48.046206 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.045546 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c47903\"/\"openshift-service-ca.crt\"" Oct 30 13:30:48.063663 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.063628 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x"] Oct 30 13:30:48.071316 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.071276 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5q8w\" (UniqueName: \"kubernetes.io/projected/1b3e9a3f-e471-4268-ac52-275511f40b7c-kube-api-access-h5q8w\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.071502 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.071454 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-tmp-dir\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.071612 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.071593 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-dshm\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.071692 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.071643 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-home\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.071761 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.071738 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b3e9a3f-e471-4268-ac52-275511f40b7c-tls-certs\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.071852 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.071827 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-model-cache\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.074248 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.074219 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c47903\"/\"auth-test-model-c47903-kserve-self-signed-certs\"" Oct 30 13:30:48.075681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.075657 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c47903\"/\"default-dockercfg-mmr45\"" Oct 30 13:30:48.173634 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.173592 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5q8w\" (UniqueName: \"kubernetes.io/projected/1b3e9a3f-e471-4268-ac52-275511f40b7c-kube-api-access-h5q8w\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.174101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.173709 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-tmp-dir\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.174101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.173754 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-dshm\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.174101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.173794 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-home\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.174101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.173850 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b3e9a3f-e471-4268-ac52-275511f40b7c-tls-certs\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.174101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.173904 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-model-cache\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.174993 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.174717 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-tmp-dir\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.174993 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.174720 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-model-cache\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.174993 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.174842 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-home\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.176935 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.176904 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-dshm\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.177438 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.177418 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b3e9a3f-e471-4268-ac52-275511f40b7c-tls-certs\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.202628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.202572 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5q8w\" (UniqueName: \"kubernetes.io/projected/1b3e9a3f-e471-4268-ac52-275511f40b7c-kube-api-access-h5q8w\") pod \"auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.364476 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.364251 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:48.714780 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:30:48.714748 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b3e9a3f_e471_4268_ac52_275511f40b7c.slice/crio-b24a8f936bb2f6c6f23c7bb60ab3908396480ac2e995f0ed8905ad4445b5b01d WatchSource:0}: Error finding container b24a8f936bb2f6c6f23c7bb60ab3908396480ac2e995f0ed8905ad4445b5b01d: Status 404 returned error can't find the container with id b24a8f936bb2f6c6f23c7bb60ab3908396480ac2e995f0ed8905ad4445b5b01d Oct 30 13:30:48.714933 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.714831 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x"] Oct 30 13:30:48.810616 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.809970 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" event={"ID":"1b3e9a3f-e471-4268-ac52-275511f40b7c","Type":"ContainerStarted","Data":"b24a8f936bb2f6c6f23c7bb60ab3908396480ac2e995f0ed8905ad4445b5b01d"} Oct 30 13:30:48.825557 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.822768 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" event={"ID":"998f2069-f1df-403c-9719-46a9a3cda3fa","Type":"ContainerStarted","Data":"bb8db0c1c99387b9a10741c73f5a99a73a73b8a70fc2504738274b80bdc4c34e"} Oct 30 13:30:48.825557 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.822839 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" event={"ID":"998f2069-f1df-403c-9719-46a9a3cda3fa","Type":"ContainerStarted","Data":"28ea46e187b0872a540d5931fa71c05f19be485462b3d323bbac3f796ed3639e"} Oct 30 13:30:48.830546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.829604 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:30:48.899765 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:48.899700 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" podStartSLOduration=1.565493301 podStartE2EDuration="1.899678339s" totalImagesPullingTime="334.185038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:48.896408797 +0000 UTC m=+2305.528595960" watchObservedRunningTime="2026-10-30 13:30:48.899678339 +0000 UTC m=+2305.531865503" Oct 30 13:30:49.830459 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:49.830412 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" event={"ID":"1b3e9a3f-e471-4268-ac52-275511f40b7c","Type":"ContainerStarted","Data":"1a3a0937377e03dc3d8f588590dd2f100dfd1ae2086623648546123745236e41"} Oct 30 13:30:49.879089 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:49.879033 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" podStartSLOduration=2.879019563 podStartE2EDuration="2.879019563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:49.875989464 +0000 UTC m=+2306.508176628" watchObservedRunningTime="2026-10-30 13:30:49.879019563 +0000 UTC m=+2306.511206722" Oct 30 13:30:50.365034 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:50.365000 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:54.836753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:54.836716 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:30:58.365733 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:58.365699 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:58.378766 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:58.378740 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:58.899319 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:30:58.899288 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:30:59.401272 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:30:59.401229 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:31:24.651889 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.651849 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf"] Oct 30 13:31:24.656262 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.656240 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.673057 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.673014 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf"] Oct 30 13:31:24.681753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.681722 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-b29df4\"/\"auth-test-model-b29df4-kserve-self-signed-certs\"" Oct 30 13:31:24.681956 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.681796 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-b29df4\"/\"default-dockercfg-dtksm\"" Oct 30 13:31:24.686221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.686190 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-b29df4\"/\"openshift-service-ca.crt\"" Oct 30 13:31:24.686363 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.686226 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-b29df4\"/\"kube-root-ca.crt\"" Oct 30 13:31:24.776412 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.776372 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-model-cache\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.776762 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.776720 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95176be7-e786-4ddc-bf3f-99ae16ff9c20-tls-certs\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.776885 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.776779 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2722\" (UniqueName: \"kubernetes.io/projected/95176be7-e786-4ddc-bf3f-99ae16ff9c20-kube-api-access-q2722\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.776958 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.776889 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-tmp-dir\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.777014 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.776950 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-dshm\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.777173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.777152 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-home\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.878688 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.878648 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-home\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.878886 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.878756 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-model-cache\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.878886 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.878818 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95176be7-e786-4ddc-bf3f-99ae16ff9c20-tls-certs\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.878886 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.878854 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2722\" (UniqueName: \"kubernetes.io/projected/95176be7-e786-4ddc-bf3f-99ae16ff9c20-kube-api-access-q2722\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.879076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.878898 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-tmp-dir\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.879076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.878944 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-dshm\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.879185 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.879107 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-home\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.879185 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.879134 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-model-cache\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.879389 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.879366 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-tmp-dir\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.881788 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.881759 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95176be7-e786-4ddc-bf3f-99ae16ff9c20-tls-certs\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.885807 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.885779 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-dshm\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.905070 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.904994 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2722\" (UniqueName: \"kubernetes.io/projected/95176be7-e786-4ddc-bf3f-99ae16ff9c20-kube-api-access-q2722\") pod \"auth-test-model-b29df4-kserve-bffb9cccc-k7tvf\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:24.976814 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:24.976774 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:25.334917 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:25.334890 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf"] Oct 30 13:31:25.336688 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:31:25.336664 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95176be7_e786_4ddc_bf3f_99ae16ff9c20.slice/crio-1328bf6e064f48c577043f52f8d216232e53164729fb6aef80061e00801c2cec WatchSource:0}: Error finding container 1328bf6e064f48c577043f52f8d216232e53164729fb6aef80061e00801c2cec: Status 404 returned error can't find the container with id 1328bf6e064f48c577043f52f8d216232e53164729fb6aef80061e00801c2cec Oct 30 13:31:26.081161 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:26.081121 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" event={"ID":"95176be7-e786-4ddc-bf3f-99ae16ff9c20","Type":"ContainerStarted","Data":"08a63d358d206f444c57f31d3075ff771b98e2d80265c63b34f1d77d12c40d51"} Oct 30 13:31:26.081161 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:26.081167 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" event={"ID":"95176be7-e786-4ddc-bf3f-99ae16ff9c20","Type":"ContainerStarted","Data":"1328bf6e064f48c577043f52f8d216232e53164729fb6aef80061e00801c2cec"} Oct 30 13:31:26.126758 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:26.126687 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" podStartSLOduration=2.126668768 podStartE2EDuration="2.126668768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:26.124374356 +0000 UTC m=+2342.756561518" watchObservedRunningTime="2026-10-30 13:31:26.126668768 +0000 UTC m=+2342.758855933" Oct 30 13:31:26.978062 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:26.978028 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:29.402941 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:31:29.402898 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:31:34.979179 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:34.979084 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:34.992702 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:34.992670 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:35.153836 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:35.153808 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:31:36.417364 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.417326 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59"] Oct 30 13:31:36.421597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.421579 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.439347 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.439321 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-5a6281-openshift-default-dockercfg-k6zpv\"" Oct 30 13:31:36.451755 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.451718 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-envoy\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.451941 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.451878 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-credential-socket\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.452073 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.452052 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-podinfo\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.452138 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.452125 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-token\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.452285 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.452265 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86738af-6a6b-428d-8141-c3c4eb82a923-istiod-ca-cert\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.452358 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.452308 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-workload-certs\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.452442 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.452423 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-workload-socket\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.452515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.452468 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-data\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.452576 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.452518 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmpzp\" (UniqueName: \"kubernetes.io/projected/b86738af-6a6b-428d-8141-c3c4eb82a923-kube-api-access-rmpzp\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.456575 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.456546 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59"] Oct 30 13:31:36.554225 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.554181 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-token\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.554225 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.554234 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86738af-6a6b-428d-8141-c3c4eb82a923-istiod-ca-cert\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.554547 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.554253 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-workload-certs\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.554547 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.554313 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-workload-socket\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.554547 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.554348 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-data\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.554547 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.554379 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmpzp\" (UniqueName: \"kubernetes.io/projected/b86738af-6a6b-428d-8141-c3c4eb82a923-kube-api-access-rmpzp\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.554547 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.554537 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-envoy\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.554782 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.554684 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-credential-socket\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.554839 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.554816 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-workload-socket\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.554904 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.554889 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-podinfo\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.554904 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.554888 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-data\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.555071 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.555043 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-workload-certs\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.555235 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.555215 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-credential-socket\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.555601 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.555570 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86738af-6a6b-428d-8141-c3c4eb82a923-istiod-ca-cert\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.556957 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.556925 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-envoy\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.557234 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.557216 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-podinfo\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.578718 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.578686 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-token\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.578942 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.578920 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmpzp\" (UniqueName: \"kubernetes.io/projected/b86738af-6a6b-428d-8141-c3c4eb82a923-kube-api-access-rmpzp\") pod \"e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:36.736795 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:36.736697 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:37.070328 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:37.070294 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59"] Oct 30 13:31:37.072201 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:31:37.072154 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86738af_6a6b_428d_8141_c3c4eb82a923.slice/crio-df9cd4bf51844c119bdad51117e323563e8fcf679ac517ef4083535bb6299860 WatchSource:0}: Error finding container df9cd4bf51844c119bdad51117e323563e8fcf679ac517ef4083535bb6299860: Status 404 returned error can't find the container with id df9cd4bf51844c119bdad51117e323563e8fcf679ac517ef4083535bb6299860 Oct 30 13:31:37.075769 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:37.075733 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:31:37.075869 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:37.075847 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:31:37.075915 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:37.075895 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:31:37.164561 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:37.164522 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" event={"ID":"b86738af-6a6b-428d-8141-c3c4eb82a923","Type":"ContainerStarted","Data":"df9cd4bf51844c119bdad51117e323563e8fcf679ac517ef4083535bb6299860"} Oct 30 13:31:38.172595 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:38.172555 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" event={"ID":"b86738af-6a6b-428d-8141-c3c4eb82a923","Type":"ContainerStarted","Data":"aaf4b2a92fb636059b23c62319b41623a2b4f67f8fff84b1baffefe4e3e62f66"} Oct 30 13:31:38.745511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:38.742778 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:38.745511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:38.742919 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 13:31:38.745511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:38.742963 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" podUID="b86738af-6a6b-428d-8141-c3c4eb82a923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 13:31:39.577513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:39.576537 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:31:39.744530 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:39.739986 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 13:31:39.744530 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:39.740050 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" podUID="b86738af-6a6b-428d-8141-c3c4eb82a923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 13:31:40.743517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:40.740398 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 13:31:40.743517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:40.740477 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" podUID="b86738af-6a6b-428d-8141-c3c4eb82a923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 13:31:41.740632 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:41.740588 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 13:31:41.740917 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:41.740886 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" podUID="b86738af-6a6b-428d-8141-c3c4eb82a923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 13:31:42.740427 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:42.739966 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 13:31:42.740427 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:42.740035 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" podUID="b86738af-6a6b-428d-8141-c3c4eb82a923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 13:31:43.003820 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:43.003690 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" podStartSLOduration=7.003668052 podStartE2EDuration="7.003668052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:38.234749307 +0000 UTC m=+2354.866936470" watchObservedRunningTime="2026-10-30 13:31:43.003668052 +0000 UTC m=+2359.635855215" Oct 30 13:31:43.054728 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:43.054673 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk"] Oct 30 13:31:43.055182 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:43.055133 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" podUID="27c0ae74-f815-4746-94d7-959ad7a2733e" containerName="istio-proxy" containerID="cri-o://ea57c142625133eea709eba14b2cf672de303eb82f3f4f2506ebb5644e16a2b8" gracePeriod=30 Oct 30 13:31:43.737168 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:43.737131 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 13:31:43.737355 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:43.737188 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" podUID="b86738af-6a6b-428d-8141-c3c4eb82a923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 13:31:44.742974 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:44.742940 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:44.743395 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:44.743372 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:44.744294 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:44.744275 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:31:46.856244 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:46.856200 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:31:46.856694 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:46.856265 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" podUID="27c0ae74-f815-4746-94d7-959ad7a2733e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:31:48.349618 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.349589 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:31:48.417054 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.417013 7010 generic.go:356] "Generic (PLEG): container finished" podID="27c0ae74-f815-4746-94d7-959ad7a2733e" containerID="ea57c142625133eea709eba14b2cf672de303eb82f3f4f2506ebb5644e16a2b8" exitCode=0 Oct 30 13:31:48.417238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.417074 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" Oct 30 13:31:48.417238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.417105 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" event={"ID":"27c0ae74-f815-4746-94d7-959ad7a2733e","Type":"ContainerDied","Data":"ea57c142625133eea709eba14b2cf672de303eb82f3f4f2506ebb5644e16a2b8"} Oct 30 13:31:48.417238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.417215 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk" event={"ID":"27c0ae74-f815-4746-94d7-959ad7a2733e","Type":"ContainerDied","Data":"a344057e0069e18ca4fb722c47f3c61d7029414417653a459925bef14c488586"} Oct 30 13:31:48.417391 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.417246 7010 scope.go:122] "RemoveContainer" containerID="ea57c142625133eea709eba14b2cf672de303eb82f3f4f2506ebb5644e16a2b8" Oct 30 13:31:48.439781 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.439696 7010 scope.go:122] "RemoveContainer" containerID="ea57c142625133eea709eba14b2cf672de303eb82f3f4f2506ebb5644e16a2b8" Oct 30 13:31:48.440123 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:31:48.440098 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea57c142625133eea709eba14b2cf672de303eb82f3f4f2506ebb5644e16a2b8\": container with ID starting with ea57c142625133eea709eba14b2cf672de303eb82f3f4f2506ebb5644e16a2b8 not found: ID does not exist" containerID="ea57c142625133eea709eba14b2cf672de303eb82f3f4f2506ebb5644e16a2b8" Oct 30 13:31:48.440223 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.440134 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea57c142625133eea709eba14b2cf672de303eb82f3f4f2506ebb5644e16a2b8"} err="failed to get container status \"ea57c142625133eea709eba14b2cf672de303eb82f3f4f2506ebb5644e16a2b8\": rpc error: code = NotFound desc = could not find container \"ea57c142625133eea709eba14b2cf672de303eb82f3f4f2506ebb5644e16a2b8\": container with ID starting with ea57c142625133eea709eba14b2cf672de303eb82f3f4f2506ebb5644e16a2b8 not found: ID does not exist" Oct 30 13:31:48.449602 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.449572 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-workload-socket\") pod \"27c0ae74-f815-4746-94d7-959ad7a2733e\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " Oct 30 13:31:48.449724 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.449705 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c0ae74-f815-4746-94d7-959ad7a2733e-kube-api-access-5xl94\" (UniqueName: \"kubernetes.io/projected/27c0ae74-f815-4746-94d7-959ad7a2733e-kube-api-access-5xl94\") pod \"27c0ae74-f815-4746-94d7-959ad7a2733e\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " Oct 30 13:31:48.449772 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.449754 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c0ae74-f815-4746-94d7-959ad7a2733e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c0ae74-f815-4746-94d7-959ad7a2733e-istiod-ca-cert\") pod \"27c0ae74-f815-4746-94d7-959ad7a2733e\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " Oct 30 13:31:48.449820 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.449805 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-envoy\") pod \"27c0ae74-f815-4746-94d7-959ad7a2733e\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " Oct 30 13:31:48.449893 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.449879 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-token\" (UniqueName: \"kubernetes.io/projected/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-token\") pod \"27c0ae74-f815-4746-94d7-959ad7a2733e\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " Oct 30 13:31:48.449933 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.449920 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-data\") pod \"27c0ae74-f815-4746-94d7-959ad7a2733e\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " Oct 30 13:31:48.450088 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.450057 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-workload-certs\") pod \"27c0ae74-f815-4746-94d7-959ad7a2733e\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " Oct 30 13:31:48.450226 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.450107 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-credential-socket\") pod \"27c0ae74-f815-4746-94d7-959ad7a2733e\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " Oct 30 13:31:48.450226 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.450137 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-podinfo\") pod \"27c0ae74-f815-4746-94d7-959ad7a2733e\" (UID: \"27c0ae74-f815-4746-94d7-959ad7a2733e\") " Oct 30 13:31:48.450357 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.450307 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c0ae74-f815-4746-94d7-959ad7a2733e-istiod-ca-cert" pod "27c0ae74-f815-4746-94d7-959ad7a2733e" (UID: "27c0ae74-f815-4746-94d7-959ad7a2733e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:31:48.450548 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.450525 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-workload-socket" pod "27c0ae74-f815-4746-94d7-959ad7a2733e" (UID: "27c0ae74-f815-4746-94d7-959ad7a2733e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:31:48.450893 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.450872 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:31:48.451037 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.451017 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-data" pod "27c0ae74-f815-4746-94d7-959ad7a2733e" (UID: "27c0ae74-f815-4746-94d7-959ad7a2733e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:31:48.451118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.451021 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c0ae74-f815-4746-94d7-959ad7a2733e-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:31:48.453407 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.453132 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c0ae74-f815-4746-94d7-959ad7a2733e-kube-api-access-5xl94" pod "27c0ae74-f815-4746-94d7-959ad7a2733e" (UID: "27c0ae74-f815-4746-94d7-959ad7a2733e"). InnerVolumeSpecName "kube-api-access-5xl94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:31:48.453407 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.453218 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-podinfo" pod "27c0ae74-f815-4746-94d7-959ad7a2733e" (UID: "27c0ae74-f815-4746-94d7-959ad7a2733e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:31:48.453599 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.453524 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-credential-socket" pod "27c0ae74-f815-4746-94d7-959ad7a2733e" (UID: "27c0ae74-f815-4746-94d7-959ad7a2733e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:31:48.453741 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.453704 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-workload-certs" pod "27c0ae74-f815-4746-94d7-959ad7a2733e" (UID: "27c0ae74-f815-4746-94d7-959ad7a2733e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:31:48.454259 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.454232 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-token" pod "27c0ae74-f815-4746-94d7-959ad7a2733e" (UID: "27c0ae74-f815-4746-94d7-959ad7a2733e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:31:48.454259 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.454242 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-envoy" pod "27c0ae74-f815-4746-94d7-959ad7a2733e" (UID: "27c0ae74-f815-4746-94d7-959ad7a2733e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:31:48.551960 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.551925 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:31:48.551960 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.551953 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:31:48.551960 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.551963 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:31:48.551960 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.551973 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xl94\" (UniqueName: \"kubernetes.io/projected/27c0ae74-f815-4746-94d7-959ad7a2733e-kube-api-access-5xl94\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:31:48.552299 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.551982 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:31:48.552299 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.551990 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:31:48.552299 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.551997 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c0ae74-f815-4746-94d7-959ad7a2733e-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:31:48.775004 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.774831 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk"] Oct 30 13:31:48.787749 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:48.787714 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-xwjnk"] Oct 30 13:31:49.425552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:49.425521 7010 generic.go:356] "Generic (PLEG): container finished" podID="fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" containerID="1f3037052e5d6bba927c98efbd6ee2226509efac2634621a8ad0f4c965d9bb20" exitCode=0 Oct 30 13:31:49.426000 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:49.425593 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" event={"ID":"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231","Type":"ContainerDied","Data":"1f3037052e5d6bba927c98efbd6ee2226509efac2634621a8ad0f4c965d9bb20"} Oct 30 13:31:49.426094 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:49.426074 7010 scope.go:122] "RemoveContainer" containerID="1f3037052e5d6bba927c98efbd6ee2226509efac2634621a8ad0f4c965d9bb20" Oct 30 13:31:49.427878 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:49.427847 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:31:49.427992 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:49.427934 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:31:49.427992 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:49.427981 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:31:49.492222 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:49.492189 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c0ae74-f815-4746-94d7-959ad7a2733e" path="/var/lib/kubelet/pods/27c0ae74-f815-4746-94d7-959ad7a2733e/volumes" Oct 30 13:31:50.262516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:50.262451 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:31:50.436077 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:50.436039 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" event={"ID":"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231","Type":"ContainerStarted","Data":"ce05764bdeb208538cf70152a554d5136ccb8828f2a2777a225919e99ab6725a"} Oct 30 13:31:51.004167 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:51.004135 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:31:51.004557 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:51.004531 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 13:31:51.004745 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:51.004721 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" podUID="fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 13:31:52.010085 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:52.010045 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 13:31:52.010613 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:52.010119 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" podUID="fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 13:31:53.007112 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:53.007069 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 13:31:53.007414 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:53.007386 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" podUID="fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 13:31:54.010309 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:54.008158 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 13:31:54.010309 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:54.008305 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" podUID="fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 13:31:55.007520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:55.004866 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 13:31:55.007520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:55.004931 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" podUID="fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 13:31:56.000855 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:56.000821 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 13:31:56.001300 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:56.000878 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" podUID="fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 13:31:57.006226 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:57.006191 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:31:57.006725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:57.006601 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:31:57.007313 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:31:57.007279 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:31:59.400766 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:31:59.400723 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:32:05.938325 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:05.938275 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s"] Oct 30 13:32:05.938908 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:05.938874 7010 cpu_manager.go:414] "containerMap: removing container" podUID="27c0ae74-f815-4746-94d7-959ad7a2733e" containerName="istio-proxy" Oct 30 13:32:05.938908 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:05.938888 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c0ae74-f815-4746-94d7-959ad7a2733e" containerName="istio-proxy" Oct 30 13:32:05.939058 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:05.939045 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c0ae74-f815-4746-94d7-959ad7a2733e" containerName="istio-proxy" Oct 30 13:32:05.949517 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:05.949303 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:05.998944 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:05.998520 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s"] Oct 30 13:32:06.022273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.021468 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.022273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.021545 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.022273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.021608 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.022273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.021671 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.022273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.021716 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.022273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.021761 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.022273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.021803 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.022273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.021899 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.022273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.021975 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr5m4\" (UniqueName: \"kubernetes.io/projected/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-kube-api-access-fr5m4\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.123099 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.123058 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.123296 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.123202 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.123353 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.123324 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.123404 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.123393 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr5m4\" (UniqueName: \"kubernetes.io/projected/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-kube-api-access-fr5m4\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.123460 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.123447 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.123563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.123527 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.123563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.123531 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-credential-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.123699 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.123628 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.123758 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.123704 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.123813 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.123753 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.124070 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.124046 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-workload-socket\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.124178 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.124135 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-workload-certs\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.124243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.124181 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-data\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.124596 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.124571 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istiod-ca-cert\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.126041 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.126017 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-envoy\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.126123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.126103 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-podinfo\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.149602 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.149570 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr5m4\" (UniqueName: \"kubernetes.io/projected/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-kube-api-access-fr5m4\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.150094 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.150057 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-token\") pod \"e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.326450 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.326361 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:06.659941 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.659907 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s"] Oct 30 13:32:06.660689 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:32:06.660660 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dafc909_0a2f_41dd_9529_76cbfad4ce5f.slice/crio-6b3e38a83d25c927d2db8337c8c3512de4a42a90efdf0499048449df69007605 WatchSource:0}: Error finding container 6b3e38a83d25c927d2db8337c8c3512de4a42a90efdf0499048449df69007605: Status 404 returned error can't find the container with id 6b3e38a83d25c927d2db8337c8c3512de4a42a90efdf0499048449df69007605 Oct 30 13:32:06.663449 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.663412 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:06.663572 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.663514 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:06.663572 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:06.663564 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:07.544315 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:07.544265 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" event={"ID":"5dafc909-0a2f-41dd-9529-76cbfad4ce5f","Type":"ContainerStarted","Data":"604ff51449344ccc20f224a52bc1b80c8a0919729e934939b300619299c20086"} Oct 30 13:32:07.544832 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:07.544331 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" event={"ID":"5dafc909-0a2f-41dd-9529-76cbfad4ce5f","Type":"ContainerStarted","Data":"6b3e38a83d25c927d2db8337c8c3512de4a42a90efdf0499048449df69007605"} Oct 30 13:32:07.607522 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:07.606275 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" podStartSLOduration=2.6062559910000003 podStartE2EDuration="2.606255991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:07.603631563 +0000 UTC m=+2384.235818751" watchObservedRunningTime="2026-10-30 13:32:07.606255991 +0000 UTC m=+2384.238443158" Oct 30 13:32:08.326571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:08.326537 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:08.328152 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:08.328126 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 13:32:08.328251 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:08.328207 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" podUID="5dafc909-0a2f-41dd-9529-76cbfad4ce5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 13:32:09.327435 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:09.327399 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 13:32:09.327950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:09.327454 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" podUID="5dafc909-0a2f-41dd-9529-76cbfad4ce5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 13:32:10.327581 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:10.327549 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 13:32:10.328022 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:10.327603 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" podUID="5dafc909-0a2f-41dd-9529-76cbfad4ce5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 13:32:11.327144 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:11.327108 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 13:32:11.327336 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:11.327162 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" podUID="5dafc909-0a2f-41dd-9529-76cbfad4ce5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 13:32:12.327389 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:12.327191 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 13:32:12.327389 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:12.327253 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" podUID="5dafc909-0a2f-41dd-9529-76cbfad4ce5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 13:32:13.332606 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:13.332574 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:13.333020 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:13.332980 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:13.333722 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:13.333701 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:32:23.903720 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:23.903690 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:32:23.914928 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:23.914903 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:32:29.401256 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:32:29.401214 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:32:41.515775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.515733 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt"] Oct 30 13:32:41.522429 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.522400 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.550097 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.541107 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-082748-openshift-default-dockercfg-fhnpr\"" Oct 30 13:32:41.560865 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.560827 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt"] Oct 30 13:32:41.599134 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.599091 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-data\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.599351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.599159 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-credential-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.599351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.599186 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-workload-certs\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.599351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.599228 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-workload-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.599351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.599254 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1589354d-4858-4ad9-a76f-305adc4c83a9-istiod-ca-cert\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.599351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.599333 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-envoy\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.599695 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.599561 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh6xm\" (UniqueName: \"kubernetes.io/projected/1589354d-4858-4ad9-a76f-305adc4c83a9-kube-api-access-kh6xm\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.599695 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.599593 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-podinfo\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.599695 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.599618 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-token\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.701152 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.701110 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-envoy\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.701373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.701313 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh6xm\" (UniqueName: \"kubernetes.io/projected/1589354d-4858-4ad9-a76f-305adc4c83a9-kube-api-access-kh6xm\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.701373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.701348 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-podinfo\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.701544 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.701374 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-token\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.701544 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.701478 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-data\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.701670 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.701542 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-credential-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.701670 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.701569 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-workload-certs\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.701670 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.701611 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-workload-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.701670 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.701637 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1589354d-4858-4ad9-a76f-305adc4c83a9-istiod-ca-cert\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.707397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.702464 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1589354d-4858-4ad9-a76f-305adc4c83a9-istiod-ca-cert\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.707397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.702825 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-workload-certs\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.741546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.731271 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-credential-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.741546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.738825 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-workload-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.741546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.740091 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-data\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.757908 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.744772 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-envoy\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.771562 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.759142 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-podinfo\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.786551 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.775402 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh6xm\" (UniqueName: \"kubernetes.io/projected/1589354d-4858-4ad9-a76f-305adc4c83a9-kube-api-access-kh6xm\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.803172 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.787080 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-token\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:41.844189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:41.841659 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:42.213481 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:42.213451 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt"] Oct 30 13:32:42.215752 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:32:42.215723 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1589354d_4858_4ad9_a76f_305adc4c83a9.slice/crio-3ab48d343f72796b4d61c5a90f69bac367f05d9ebf9b724f55b30f31952a4284 WatchSource:0}: Error finding container 3ab48d343f72796b4d61c5a90f69bac367f05d9ebf9b724f55b30f31952a4284: Status 404 returned error can't find the container with id 3ab48d343f72796b4d61c5a90f69bac367f05d9ebf9b724f55b30f31952a4284 Oct 30 13:32:42.219048 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:42.219014 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:42.219142 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:42.219084 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:42.219142 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:42.219114 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:32:42.863291 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:42.863252 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" event={"ID":"1589354d-4858-4ad9-a76f-305adc4c83a9","Type":"ContainerStarted","Data":"96eb82a94d4f48ae413198095b2aeede03c013b977864e0efff8d18c2d68dcf1"} Oct 30 13:32:42.863753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:42.863316 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" event={"ID":"1589354d-4858-4ad9-a76f-305adc4c83a9","Type":"ContainerStarted","Data":"3ab48d343f72796b4d61c5a90f69bac367f05d9ebf9b724f55b30f31952a4284"} Oct 30 13:32:42.926435 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:42.926377 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" podStartSLOduration=1.926362065 podStartE2EDuration="1.926362065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:42.911512286 +0000 UTC m=+2419.543699449" watchObservedRunningTime="2026-10-30 13:32:42.926362065 +0000 UTC m=+2419.558549226" Oct 30 13:32:43.847536 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:43.844648 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:43.860788 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:43.860749 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:43.870827 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:43.870793 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:43.872047 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:43.872021 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:32:47.533788 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.533749 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x"] Oct 30 13:32:47.538576 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.538551 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" Oct 30 13:32:47.556617 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.556592 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-082748\"" Oct 30 13:32:47.568563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.568534 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-082748-dockercfg-4xxjc\"" Oct 30 13:32:47.587117 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.587083 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x"] Oct 30 13:32:47.590354 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.590324 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc"] Oct 30 13:32:47.601744 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.601700 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" Oct 30 13:32:47.636272 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.636237 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc"] Oct 30 13:32:47.738877 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.738840 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79da1b72-a631-4475-b80e-ba86dd683b51-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-082748-74688b9686-tptfc\" (UID: \"79da1b72-a631-4475-b80e-ba86dd683b51\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" Oct 30 13:32:47.739227 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.739201 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89h5k\" (UniqueName: \"kubernetes.io/projected/04312d72-72de-4fad-b410-eeaea20664f3-kube-api-access-89h5k\") pod \"payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x\" (UID: \"04312d72-72de-4fad-b410-eeaea20664f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" Oct 30 13:32:47.739412 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.739394 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04312d72-72de-4fad-b410-eeaea20664f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x\" (UID: \"04312d72-72de-4fad-b410-eeaea20664f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" Oct 30 13:32:47.739560 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.739542 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwdf4\" (UniqueName: \"kubernetes.io/projected/79da1b72-a631-4475-b80e-ba86dd683b51-kube-api-access-rwdf4\") pod \"payload-processing-e2e-ipp-a-082748-74688b9686-tptfc\" (UID: \"79da1b72-a631-4475-b80e-ba86dd683b51\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" Oct 30 13:32:47.841597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.841482 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04312d72-72de-4fad-b410-eeaea20664f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x\" (UID: \"04312d72-72de-4fad-b410-eeaea20664f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" Oct 30 13:32:47.841597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.841559 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwdf4\" (UniqueName: \"kubernetes.io/projected/79da1b72-a631-4475-b80e-ba86dd683b51-kube-api-access-rwdf4\") pod \"payload-processing-e2e-ipp-a-082748-74688b9686-tptfc\" (UID: \"79da1b72-a631-4475-b80e-ba86dd683b51\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" Oct 30 13:32:47.841829 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.841690 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79da1b72-a631-4475-b80e-ba86dd683b51-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-082748-74688b9686-tptfc\" (UID: \"79da1b72-a631-4475-b80e-ba86dd683b51\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" Oct 30 13:32:47.841894 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.841865 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89h5k\" (UniqueName: \"kubernetes.io/projected/04312d72-72de-4fad-b410-eeaea20664f3-kube-api-access-89h5k\") pod \"payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x\" (UID: \"04312d72-72de-4fad-b410-eeaea20664f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" Oct 30 13:32:47.842502 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.842466 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04312d72-72de-4fad-b410-eeaea20664f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x\" (UID: \"04312d72-72de-4fad-b410-eeaea20664f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" Oct 30 13:32:47.842877 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.842856 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79da1b72-a631-4475-b80e-ba86dd683b51-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-082748-74688b9686-tptfc\" (UID: \"79da1b72-a631-4475-b80e-ba86dd683b51\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" Oct 30 13:32:47.866695 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.866665 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89h5k\" (UniqueName: \"kubernetes.io/projected/04312d72-72de-4fad-b410-eeaea20664f3-kube-api-access-89h5k\") pod \"payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x\" (UID: \"04312d72-72de-4fad-b410-eeaea20664f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" Oct 30 13:32:47.867379 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.867352 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwdf4\" (UniqueName: \"kubernetes.io/projected/79da1b72-a631-4475-b80e-ba86dd683b51-kube-api-access-rwdf4\") pod \"payload-processing-e2e-ipp-a-082748-74688b9686-tptfc\" (UID: \"79da1b72-a631-4475-b80e-ba86dd683b51\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" Oct 30 13:32:47.952087 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:47.952049 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" Oct 30 13:32:48.153371 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:48.153336 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" Oct 30 13:32:48.276051 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:48.275992 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc"] Oct 30 13:32:48.469920 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:48.469895 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x"] Oct 30 13:32:48.473089 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:32:48.472484 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04312d72_72de_4fad_b410_eeaea20664f3.slice/crio-930a3b7a0609cc75d448d7e6df65c74edfc45ece1b2a04e8c4bb8f4ecc54fd2f WatchSource:0}: Error finding container 930a3b7a0609cc75d448d7e6df65c74edfc45ece1b2a04e8c4bb8f4ecc54fd2f: Status 404 returned error can't find the container with id 930a3b7a0609cc75d448d7e6df65c74edfc45ece1b2a04e8c4bb8f4ecc54fd2f Oct 30 13:32:48.911218 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:48.911182 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" event={"ID":"04312d72-72de-4fad-b410-eeaea20664f3","Type":"ContainerStarted","Data":"1a28998fb1b343078ceb3b2cb3adeaa0a15f713a2eb039f2bc1ac3120fd4e823"} Oct 30 13:32:48.911677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:48.911237 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" event={"ID":"04312d72-72de-4fad-b410-eeaea20664f3","Type":"ContainerStarted","Data":"930a3b7a0609cc75d448d7e6df65c74edfc45ece1b2a04e8c4bb8f4ecc54fd2f"} Oct 30 13:32:48.911677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:48.911280 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" Oct 30 13:32:48.913197 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:48.913168 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" event={"ID":"79da1b72-a631-4475-b80e-ba86dd683b51","Type":"ContainerStarted","Data":"da40321b0375d62b2a87e125d393ec358c1df041635dcf7f0b1d1b575d7d345b"} Oct 30 13:32:48.913360 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:48.913216 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" event={"ID":"79da1b72-a631-4475-b80e-ba86dd683b51","Type":"ContainerStarted","Data":"427a1e14af16762fea7b1bd68a6ea7ba5894de4fe206fdb0207b14306463ec97"} Oct 30 13:32:48.913360 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:48.913242 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" Oct 30 13:32:48.953375 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:48.953309 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" podStartSLOduration=1.953296176 podStartE2EDuration="1.953296176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:48.952616281 +0000 UTC m=+2425.584803466" watchObservedRunningTime="2026-10-30 13:32:48.953296176 +0000 UTC m=+2425.585483339" Oct 30 13:32:48.996049 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:48.995997 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" podStartSLOduration=1.995984395 podStartE2EDuration="1.995984395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:48.994340659 +0000 UTC m=+2425.626527863" watchObservedRunningTime="2026-10-30 13:32:48.995984395 +0000 UTC m=+2425.628171557" Oct 30 13:32:59.401521 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:32:59.401451 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:32:59.922768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:59.922741 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" Oct 30 13:32:59.922968 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:32:59.922807 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" Oct 30 13:33:29.406128 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:33:29.406087 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:33:38.116535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.116499 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv"] Oct 30 13:33:38.121025 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.121000 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.143795 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.143761 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-f0a31b-openshift-default-dockercfg-txvm6\"" Oct 30 13:33:38.155728 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.155689 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv"] Oct 30 13:33:38.237259 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.237215 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzpwd\" (UniqueName: \"kubernetes.io/projected/8b9f8710-c8e9-4c17-ac76-0e511f952916-kube-api-access-jzpwd\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.237477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.237346 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-token\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.237477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.237415 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-envoy\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.237477 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.237450 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-podinfo\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.237706 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.237583 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-data\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.237706 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.237699 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-workload-socket\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.237855 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.237838 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-credential-socket\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.237902 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.237880 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9f8710-c8e9-4c17-ac76-0e511f952916-istiod-ca-cert\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.237989 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.237973 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-workload-certs\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.339828 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.339778 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-data\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.340044 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.339876 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-workload-socket\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.340044 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.339979 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-credential-socket\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.340044 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.340026 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9f8710-c8e9-4c17-ac76-0e511f952916-istiod-ca-cert\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.340212 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.340079 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-workload-certs\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.340273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.340224 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-data\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.340273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.340226 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzpwd\" (UniqueName: \"kubernetes.io/projected/8b9f8710-c8e9-4c17-ac76-0e511f952916-kube-api-access-jzpwd\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.340387 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.340304 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-workload-socket\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.340387 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.340317 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-token\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.340387 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.340344 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-credential-socket\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.340387 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.340369 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-envoy\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.340650 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.340408 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-podinfo\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.340650 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.340476 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-workload-certs\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.340794 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.340771 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9f8710-c8e9-4c17-ac76-0e511f952916-istiod-ca-cert\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.342746 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.342722 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-envoy\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.342976 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.342959 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-podinfo\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.364741 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.364709 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-token\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.364930 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.364908 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzpwd\" (UniqueName: \"kubernetes.io/projected/8b9f8710-c8e9-4c17-ac76-0e511f952916-kube-api-access-jzpwd\") pod \"e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.436079 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.436045 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:38.770826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.770791 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv"] Oct 30 13:33:38.772910 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:33:38.772878 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9f8710_c8e9_4c17_ac76_0e511f952916.slice/crio-a25ccbb4515a48fcaf02de2660a4f2b1b8acf68770f824caa684887ba64175c6 WatchSource:0}: Error finding container a25ccbb4515a48fcaf02de2660a4f2b1b8acf68770f824caa684887ba64175c6: Status 404 returned error can't find the container with id a25ccbb4515a48fcaf02de2660a4f2b1b8acf68770f824caa684887ba64175c6 Oct 30 13:33:38.775765 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.775733 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:38.775864 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.775806 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:38.775864 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:38.775837 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:33:39.338382 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:39.338332 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" event={"ID":"8b9f8710-c8e9-4c17-ac76-0e511f952916","Type":"ContainerStarted","Data":"8ddd5e029357c181ffbd085859cd63cd77574ad8b7f50abd1e584880907d5cac"} Oct 30 13:33:39.338980 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:39.338457 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" event={"ID":"8b9f8710-c8e9-4c17-ac76-0e511f952916","Type":"ContainerStarted","Data":"a25ccbb4515a48fcaf02de2660a4f2b1b8acf68770f824caa684887ba64175c6"} Oct 30 13:33:39.403134 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:39.403061 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" podStartSLOduration=1.403039195 podStartE2EDuration="1.403039195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:39.387279659 +0000 UTC m=+2476.019466820" watchObservedRunningTime="2026-10-30 13:33:39.403039195 +0000 UTC m=+2476.035226358" Oct 30 13:33:39.439511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:39.436526 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:39.439511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:39.438239 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 13:33:39.439511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:39.438301 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" podUID="8b9f8710-c8e9-4c17-ac76-0e511f952916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 13:33:40.437238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:40.437199 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 13:33:40.437789 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:40.437251 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" podUID="8b9f8710-c8e9-4c17-ac76-0e511f952916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 13:33:41.436565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:41.436521 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 13:33:41.436764 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:41.436585 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" podUID="8b9f8710-c8e9-4c17-ac76-0e511f952916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 13:33:42.436693 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:42.436639 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 13:33:42.437210 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:42.436712 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" podUID="8b9f8710-c8e9-4c17-ac76-0e511f952916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 13:33:43.436996 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:43.436960 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 13:33:43.437643 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:43.437016 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" podUID="8b9f8710-c8e9-4c17-ac76-0e511f952916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 13:33:43.991509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:43.989357 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8749-fbd54c"] Oct 30 13:33:43.997434 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:43.997405 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8749-fbd54c" Oct 30 13:33:44.027033 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:44.023872 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nqkjx\"" Oct 30 13:33:44.034027 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:44.033053 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8749-fbd54c"] Oct 30 13:33:44.127265 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:44.127224 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j5kv\" (UniqueName: \"kubernetes.io/projected/05faaf88-7378-4533-a956-2c39954287e6-kube-api-access-4j5kv\") pod \"test-curl-8749-fbd54c\" (UID: \"05faaf88-7378-4533-a956-2c39954287e6\") " pod="opendatahub/test-curl-8749-fbd54c" Oct 30 13:33:44.228300 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:44.228260 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j5kv\" (UniqueName: \"kubernetes.io/projected/05faaf88-7378-4533-a956-2c39954287e6-kube-api-access-4j5kv\") pod \"test-curl-8749-fbd54c\" (UID: \"05faaf88-7378-4533-a956-2c39954287e6\") " pod="opendatahub/test-curl-8749-fbd54c" Oct 30 13:33:44.254156 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:44.254061 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j5kv\" (UniqueName: \"kubernetes.io/projected/05faaf88-7378-4533-a956-2c39954287e6-kube-api-access-4j5kv\") pod \"test-curl-8749-fbd54c\" (UID: \"05faaf88-7378-4533-a956-2c39954287e6\") " pod="opendatahub/test-curl-8749-fbd54c" Oct 30 13:33:44.337061 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:44.337024 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8749-fbd54c" Oct 30 13:33:44.438558 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:44.438239 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 13:33:44.438558 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:44.438292 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" podUID="8b9f8710-c8e9-4c17-ac76-0e511f952916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 13:33:44.712968 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:44.712932 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8749-fbd54c"] Oct 30 13:33:45.390127 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:45.390086 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8749-fbd54c" event={"ID":"05faaf88-7378-4533-a956-2c39954287e6","Type":"ContainerStarted","Data":"29e59274e08e8e8e99c26605fa55db83391d79e70f20c668f16618a7fedfbac5"} Oct 30 13:33:45.390320 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:45.390139 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8749-fbd54c" event={"ID":"05faaf88-7378-4533-a956-2c39954287e6","Type":"ContainerStarted","Data":"ce34f26d1df31fdfe3bccae37c9e516b3df1b91a2a3378b50773424b4a7915f0"} Oct 30 13:33:45.431215 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:45.431160 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8749-fbd54c" podStartSLOduration=2.431147141 podStartE2EDuration="2.431147141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:45.431009325 +0000 UTC m=+2482.063196487" watchObservedRunningTime="2026-10-30 13:33:45.431147141 +0000 UTC m=+2482.063334350" Oct 30 13:33:45.436982 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:45.436947 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 13:33:45.437078 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:45.437011 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" podUID="8b9f8710-c8e9-4c17-ac76-0e511f952916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 13:33:46.302892 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:46.302854 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8749-fbd54c"] Oct 30 13:33:46.320687 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:46.320650 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8749-fbd54c"] Oct 30 13:33:46.442913 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:46.442879 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:46.443325 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:46.443296 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:46.444024 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:46.444006 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:33:47.403379 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:47.403315 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8749-fbd54c" podUID="05faaf88-7378-4533-a956-2c39954287e6" containerName="test-curl-8749-fbd54c" containerID="cri-o://29e59274e08e8e8e99c26605fa55db83391d79e70f20c668f16618a7fedfbac5" gracePeriod=2 Oct 30 13:33:47.419017 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:33:47.418972 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8749-fbd54c\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="05faaf88-7378-4533-a956-2c39954287e6" pod="opendatahub/test-curl-8749-fbd54c" Oct 30 13:33:49.621756 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:49.621723 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8749-fbd54c" Oct 30 13:33:49.711263 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:49.711177 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05faaf88-7378-4533-a956-2c39954287e6-kube-api-access-4j5kv\" (UniqueName: \"kubernetes.io/projected/05faaf88-7378-4533-a956-2c39954287e6-kube-api-access-4j5kv\") pod \"05faaf88-7378-4533-a956-2c39954287e6\" (UID: \"05faaf88-7378-4533-a956-2c39954287e6\") " Oct 30 13:33:49.713416 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:49.713385 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05faaf88-7378-4533-a956-2c39954287e6-kube-api-access-4j5kv" pod "05faaf88-7378-4533-a956-2c39954287e6" (UID: "05faaf88-7378-4533-a956-2c39954287e6"). InnerVolumeSpecName "kube-api-access-4j5kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:33:49.812741 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:49.812700 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j5kv\" (UniqueName: \"kubernetes.io/projected/05faaf88-7378-4533-a956-2c39954287e6-kube-api-access-4j5kv\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:33:50.424104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:50.424071 7010 generic.go:356] "Generic (PLEG): container finished" podID="05faaf88-7378-4533-a956-2c39954287e6" containerID="29e59274e08e8e8e99c26605fa55db83391d79e70f20c668f16618a7fedfbac5" exitCode=137 Oct 30 13:33:50.424294 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:50.424110 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8749-fbd54c" Oct 30 13:33:50.424294 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:50.424167 7010 scope.go:122] "RemoveContainer" containerID="29e59274e08e8e8e99c26605fa55db83391d79e70f20c668f16618a7fedfbac5" Oct 30 13:33:50.435787 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:50.435767 7010 scope.go:122] "RemoveContainer" containerID="29e59274e08e8e8e99c26605fa55db83391d79e70f20c668f16618a7fedfbac5" Oct 30 13:33:50.436121 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:33:50.436095 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e59274e08e8e8e99c26605fa55db83391d79e70f20c668f16618a7fedfbac5\": container with ID starting with 29e59274e08e8e8e99c26605fa55db83391d79e70f20c668f16618a7fedfbac5 not found: ID does not exist" containerID="29e59274e08e8e8e99c26605fa55db83391d79e70f20c668f16618a7fedfbac5" Oct 30 13:33:50.436217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:50.436130 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e59274e08e8e8e99c26605fa55db83391d79e70f20c668f16618a7fedfbac5"} err="failed to get container status \"29e59274e08e8e8e99c26605fa55db83391d79e70f20c668f16618a7fedfbac5\": rpc error: code = NotFound desc = could not find container \"29e59274e08e8e8e99c26605fa55db83391d79e70f20c668f16618a7fedfbac5\": container with ID starting with 29e59274e08e8e8e99c26605fa55db83391d79e70f20c668f16618a7fedfbac5 not found: ID does not exist" Oct 30 13:33:50.447148 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:33:50.447114 7010 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8749-fbd54c\" is forbidden: User \"system:node:ip-10-0-138-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-103.ec2.internal' and this object" podUID="05faaf88-7378-4533-a956-2c39954287e6" pod="opendatahub/test-curl-8749-fbd54c" Oct 30 13:33:51.501014 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:51.500973 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05faaf88-7378-4533-a956-2c39954287e6" path="/var/lib/kubelet/pods/05faaf88-7378-4533-a956-2c39954287e6/volumes" Oct 30 13:33:59.400943 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:33:59.400902 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:33:59.790884 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:59.790804 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt"] Oct 30 13:33:59.791414 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:59.791401 7010 cpu_manager.go:414] "containerMap: removing container" podUID="05faaf88-7378-4533-a956-2c39954287e6" containerName="test-curl-8749-fbd54c" Oct 30 13:33:59.791460 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:59.791416 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05faaf88-7378-4533-a956-2c39954287e6" containerName="test-curl-8749-fbd54c" Oct 30 13:33:59.791563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:59.791553 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="05faaf88-7378-4533-a956-2c39954287e6" containerName="test-curl-8749-fbd54c" Oct 30 13:33:59.796448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:59.796426 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:33:59.812959 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:59.812940 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-4b0ea9-openshift-default-dockercfg-wbmk9\"" Oct 30 13:33:59.829107 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:59.829078 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt"] Oct 30 13:33:59.959480 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:59.959447 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-workload-certs\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:33:59.959709 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:59.959525 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-data\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:33:59.959709 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:59.959621 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-workload-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:33:59.959709 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:59.959675 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f46ced9-2698-442f-8e00-1f7766b92d3c-istiod-ca-cert\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:33:59.959841 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:59.959799 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85ml5\" (UniqueName: \"kubernetes.io/projected/3f46ced9-2698-442f-8e00-1f7766b92d3c-kube-api-access-85ml5\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:33:59.959916 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:59.959898 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-credential-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:33:59.959982 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:59.959970 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-envoy\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:33:59.960039 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:59.960027 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-podinfo\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:33:59.960074 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:33:59.960052 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-token\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.061332 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.061240 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-podinfo\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.061332 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.061290 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-token\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.061651 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.061360 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-workload-certs\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.061651 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.061410 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-data\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.061651 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.061447 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-workload-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.061651 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.061504 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f46ced9-2698-442f-8e00-1f7766b92d3c-istiod-ca-cert\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.061651 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.061607 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85ml5\" (UniqueName: \"kubernetes.io/projected/3f46ced9-2698-442f-8e00-1f7766b92d3c-kube-api-access-85ml5\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.061928 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.061700 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-credential-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.061928 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.061744 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-envoy\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.061928 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.061902 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-data\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.062065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.061976 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-workload-certs\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.062383 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.062354 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f46ced9-2698-442f-8e00-1f7766b92d3c-istiod-ca-cert\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.062520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.062399 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-workload-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.062520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.062364 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-credential-socket\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.064139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.064120 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-podinfo\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.064223 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.064147 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-envoy\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.085390 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.085367 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-token\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.085523 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.085441 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85ml5\" (UniqueName: \"kubernetes.io/projected/3f46ced9-2698-442f-8e00-1f7766b92d3c-kube-api-access-85ml5\") pod \"e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.111155 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.111125 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:00.439781 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.439744 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt"] Oct 30 13:34:00.442032 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:34:00.442003 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f46ced9_2698_442f_8e00_1f7766b92d3c.slice/crio-8cbd0148ecdf30e4803108c42001851baf0d2473d63ab191779206c92e9e3168 WatchSource:0}: Error finding container 8cbd0148ecdf30e4803108c42001851baf0d2473d63ab191779206c92e9e3168: Status 404 returned error can't find the container with id 8cbd0148ecdf30e4803108c42001851baf0d2473d63ab191779206c92e9e3168 Oct 30 13:34:00.445347 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.445308 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:34:00.445464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.445397 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:34:00.445464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.445452 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:34:00.496609 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:00.496580 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" event={"ID":"3f46ced9-2698-442f-8e00-1f7766b92d3c","Type":"ContainerStarted","Data":"8cbd0148ecdf30e4803108c42001851baf0d2473d63ab191779206c92e9e3168"} Oct 30 13:34:01.516564 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:01.516519 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" event={"ID":"3f46ced9-2698-442f-8e00-1f7766b92d3c","Type":"ContainerStarted","Data":"82dc6d099214a6d41c42568e16a51249f70d1007809b594895cfbf09dc7e1518"} Oct 30 13:34:01.580253 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:01.580197 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" podStartSLOduration=2.580181248 podStartE2EDuration="2.580181248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:01.575805253 +0000 UTC m=+2498.207992417" watchObservedRunningTime="2026-10-30 13:34:01.580181248 +0000 UTC m=+2498.212368410" Oct 30 13:34:02.111898 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:02.111857 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:02.113506 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:02.113457 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 13:34:02.113687 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:02.113556 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" podUID="3f46ced9-2698-442f-8e00-1f7766b92d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 13:34:03.112396 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:03.112357 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 13:34:03.112881 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:03.112419 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" podUID="3f46ced9-2698-442f-8e00-1f7766b92d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 13:34:04.111959 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:04.111907 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 13:34:04.112185 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:04.111976 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" podUID="3f46ced9-2698-442f-8e00-1f7766b92d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 13:34:05.114942 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:05.114897 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 13:34:05.115537 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:05.115472 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" podUID="3f46ced9-2698-442f-8e00-1f7766b92d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 13:34:06.112057 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:06.112018 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 13:34:06.112242 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:06.112090 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" podUID="3f46ced9-2698-442f-8e00-1f7766b92d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 13:34:07.117641 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:07.117609 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:07.118131 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:07.117921 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:07.118652 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:07.118630 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:34:17.585409 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:17.585366 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc"] Oct 30 13:34:17.617528 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:17.617481 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x"] Oct 30 13:34:22.586202 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:22.586161 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" podUID="79da1b72-a631-4475-b80e-ba86dd683b51" containerName="payload-processing" containerID="cri-o://da40321b0375d62b2a87e125d393ec358c1df041635dcf7f0b1d1b575d7d345b" gracePeriod=25 Oct 30 13:34:22.617820 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:22.617775 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" podUID="04312d72-72de-4fad-b410-eeaea20664f3" containerName="payload-pre-processing" containerID="cri-o://1a28998fb1b343078ceb3b2cb3adeaa0a15f713a2eb039f2bc1ac3120fd4e823" gracePeriod=25 Oct 30 13:34:22.909241 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:22.909215 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" Oct 30 13:34:22.927790 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:22.927762 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" Oct 30 13:34:22.958878 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:22.958837 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04312d72-72de-4fad-b410-eeaea20664f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04312d72-72de-4fad-b410-eeaea20664f3-plugins-config-volume\") pod \"04312d72-72de-4fad-b410-eeaea20664f3\" (UID: \"04312d72-72de-4fad-b410-eeaea20664f3\") " Oct 30 13:34:22.959066 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:22.958925 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79da1b72-a631-4475-b80e-ba86dd683b51-kube-api-access-rwdf4\" (UniqueName: \"kubernetes.io/projected/79da1b72-a631-4475-b80e-ba86dd683b51-kube-api-access-rwdf4\") pod \"79da1b72-a631-4475-b80e-ba86dd683b51\" (UID: \"79da1b72-a631-4475-b80e-ba86dd683b51\") " Oct 30 13:34:22.959066 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:22.959037 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79da1b72-a631-4475-b80e-ba86dd683b51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79da1b72-a631-4475-b80e-ba86dd683b51-plugins-config-volume\") pod \"79da1b72-a631-4475-b80e-ba86dd683b51\" (UID: \"79da1b72-a631-4475-b80e-ba86dd683b51\") " Oct 30 13:34:22.959277 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:22.959257 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04312d72-72de-4fad-b410-eeaea20664f3-kube-api-access-89h5k\" (UniqueName: \"kubernetes.io/projected/04312d72-72de-4fad-b410-eeaea20664f3-kube-api-access-89h5k\") pod \"04312d72-72de-4fad-b410-eeaea20664f3\" (UID: \"04312d72-72de-4fad-b410-eeaea20664f3\") " Oct 30 13:34:22.959394 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:22.959369 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04312d72-72de-4fad-b410-eeaea20664f3-plugins-config-volume" pod "04312d72-72de-4fad-b410-eeaea20664f3" (UID: "04312d72-72de-4fad-b410-eeaea20664f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:34:22.959458 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:22.959418 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79da1b72-a631-4475-b80e-ba86dd683b51-plugins-config-volume" pod "79da1b72-a631-4475-b80e-ba86dd683b51" (UID: "79da1b72-a631-4475-b80e-ba86dd683b51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:34:22.959879 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:22.959858 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04312d72-72de-4fad-b410-eeaea20664f3-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:34:22.959879 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:22.959880 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79da1b72-a631-4475-b80e-ba86dd683b51-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:34:22.961655 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:22.961631 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04312d72-72de-4fad-b410-eeaea20664f3-kube-api-access-89h5k" pod "04312d72-72de-4fad-b410-eeaea20664f3" (UID: "04312d72-72de-4fad-b410-eeaea20664f3"). InnerVolumeSpecName "kube-api-access-89h5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:22.961932 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:22.961910 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79da1b72-a631-4475-b80e-ba86dd683b51-kube-api-access-rwdf4" pod "79da1b72-a631-4475-b80e-ba86dd683b51" (UID: "79da1b72-a631-4475-b80e-ba86dd683b51"). InnerVolumeSpecName "kube-api-access-rwdf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:23.061595 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.061561 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwdf4\" (UniqueName: \"kubernetes.io/projected/79da1b72-a631-4475-b80e-ba86dd683b51-kube-api-access-rwdf4\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:34:23.061595 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.061590 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89h5k\" (UniqueName: \"kubernetes.io/projected/04312d72-72de-4fad-b410-eeaea20664f3-kube-api-access-89h5k\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:34:23.677599 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.677558 7010 generic.go:356] "Generic (PLEG): container finished" podID="04312d72-72de-4fad-b410-eeaea20664f3" containerID="1a28998fb1b343078ceb3b2cb3adeaa0a15f713a2eb039f2bc1ac3120fd4e823" exitCode=0 Oct 30 13:34:23.678018 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.677626 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" Oct 30 13:34:23.678018 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.677633 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" event={"ID":"04312d72-72de-4fad-b410-eeaea20664f3","Type":"ContainerDied","Data":"1a28998fb1b343078ceb3b2cb3adeaa0a15f713a2eb039f2bc1ac3120fd4e823"} Oct 30 13:34:23.678018 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.677682 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x" event={"ID":"04312d72-72de-4fad-b410-eeaea20664f3","Type":"ContainerDied","Data":"930a3b7a0609cc75d448d7e6df65c74edfc45ece1b2a04e8c4bb8f4ecc54fd2f"} Oct 30 13:34:23.678018 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.677709 7010 scope.go:122] "RemoveContainer" containerID="1a28998fb1b343078ceb3b2cb3adeaa0a15f713a2eb039f2bc1ac3120fd4e823" Oct 30 13:34:23.679100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.679076 7010 generic.go:356] "Generic (PLEG): container finished" podID="79da1b72-a631-4475-b80e-ba86dd683b51" containerID="da40321b0375d62b2a87e125d393ec358c1df041635dcf7f0b1d1b575d7d345b" exitCode=0 Oct 30 13:34:23.679173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.679105 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" event={"ID":"79da1b72-a631-4475-b80e-ba86dd683b51","Type":"ContainerDied","Data":"da40321b0375d62b2a87e125d393ec358c1df041635dcf7f0b1d1b575d7d345b"} Oct 30 13:34:23.679173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.679135 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" Oct 30 13:34:23.679173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.679154 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc" event={"ID":"79da1b72-a631-4475-b80e-ba86dd683b51","Type":"ContainerDied","Data":"427a1e14af16762fea7b1bd68a6ea7ba5894de4fe206fdb0207b14306463ec97"} Oct 30 13:34:23.688324 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.688297 7010 scope.go:122] "RemoveContainer" containerID="1a28998fb1b343078ceb3b2cb3adeaa0a15f713a2eb039f2bc1ac3120fd4e823" Oct 30 13:34:23.688601 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:34:23.688580 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a28998fb1b343078ceb3b2cb3adeaa0a15f713a2eb039f2bc1ac3120fd4e823\": container with ID starting with 1a28998fb1b343078ceb3b2cb3adeaa0a15f713a2eb039f2bc1ac3120fd4e823 not found: ID does not exist" containerID="1a28998fb1b343078ceb3b2cb3adeaa0a15f713a2eb039f2bc1ac3120fd4e823" Oct 30 13:34:23.688691 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.688611 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a28998fb1b343078ceb3b2cb3adeaa0a15f713a2eb039f2bc1ac3120fd4e823"} err="failed to get container status \"1a28998fb1b343078ceb3b2cb3adeaa0a15f713a2eb039f2bc1ac3120fd4e823\": rpc error: code = NotFound desc = could not find container \"1a28998fb1b343078ceb3b2cb3adeaa0a15f713a2eb039f2bc1ac3120fd4e823\": container with ID starting with 1a28998fb1b343078ceb3b2cb3adeaa0a15f713a2eb039f2bc1ac3120fd4e823 not found: ID does not exist" Oct 30 13:34:23.688691 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.688634 7010 scope.go:122] "RemoveContainer" containerID="da40321b0375d62b2a87e125d393ec358c1df041635dcf7f0b1d1b575d7d345b" Oct 30 13:34:23.698574 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.698558 7010 scope.go:122] "RemoveContainer" containerID="da40321b0375d62b2a87e125d393ec358c1df041635dcf7f0b1d1b575d7d345b" Oct 30 13:34:23.698847 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:34:23.698829 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da40321b0375d62b2a87e125d393ec358c1df041635dcf7f0b1d1b575d7d345b\": container with ID starting with da40321b0375d62b2a87e125d393ec358c1df041635dcf7f0b1d1b575d7d345b not found: ID does not exist" containerID="da40321b0375d62b2a87e125d393ec358c1df041635dcf7f0b1d1b575d7d345b" Oct 30 13:34:23.698897 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.698855 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da40321b0375d62b2a87e125d393ec358c1df041635dcf7f0b1d1b575d7d345b"} err="failed to get container status \"da40321b0375d62b2a87e125d393ec358c1df041635dcf7f0b1d1b575d7d345b\": rpc error: code = NotFound desc = could not find container \"da40321b0375d62b2a87e125d393ec358c1df041635dcf7f0b1d1b575d7d345b\": container with ID starting with da40321b0375d62b2a87e125d393ec358c1df041635dcf7f0b1d1b575d7d345b not found: ID does not exist" Oct 30 13:34:23.731954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.731934 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc"] Oct 30 13:34:23.744371 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.744348 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-082748-74688b9686-tptfc"] Oct 30 13:34:23.776011 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.775949 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x"] Oct 30 13:34:23.780715 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:23.780693 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-78fcf548ff-2m92x"] Oct 30 13:34:25.489920 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:25.489837 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04312d72-72de-4fad-b410-eeaea20664f3" path="/var/lib/kubelet/pods/04312d72-72de-4fad-b410-eeaea20664f3/volumes" Oct 30 13:34:25.490448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:25.490428 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79da1b72-a631-4475-b80e-ba86dd683b51" path="/var/lib/kubelet/pods/79da1b72-a631-4475-b80e-ba86dd683b51/volumes" Oct 30 13:34:29.399863 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:34:29.399823 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:34:35.555256 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:35.555056 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x"] Oct 30 13:34:41.965863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:41.965826 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9"] Oct 30 13:34:41.966666 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:41.966642 7010 cpu_manager.go:414] "containerMap: removing container" podUID="79da1b72-a631-4475-b80e-ba86dd683b51" containerName="payload-processing" Oct 30 13:34:41.966784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:41.966666 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79da1b72-a631-4475-b80e-ba86dd683b51" containerName="payload-processing" Oct 30 13:34:41.966784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:41.966736 7010 cpu_manager.go:414] "containerMap: removing container" podUID="04312d72-72de-4fad-b410-eeaea20664f3" containerName="payload-pre-processing" Oct 30 13:34:41.966784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:41.966745 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04312d72-72de-4fad-b410-eeaea20664f3" containerName="payload-pre-processing" Oct 30 13:34:41.966950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:41.966902 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="04312d72-72de-4fad-b410-eeaea20664f3" containerName="payload-pre-processing" Oct 30 13:34:41.966950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:41.966924 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="79da1b72-a631-4475-b80e-ba86dd683b51" containerName="payload-processing" Oct 30 13:34:41.972369 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:41.972343 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.019977 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.019771 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9"] Oct 30 13:34:42.050546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.050164 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-data\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.050546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.050265 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-workload-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.050546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.050333 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-credential-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.050546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.050397 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-envoy\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.050546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.050433 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-podinfo\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.050546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.050517 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915ff449-5d83-4605-9ab8-e2ba528028a8-istiod-ca-cert\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.050546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.050546 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-workload-certs\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.051021 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.050596 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-token\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.051021 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.050676 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhm7s\" (UniqueName: \"kubernetes.io/projected/915ff449-5d83-4605-9ab8-e2ba528028a8-kube-api-access-vhm7s\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.152850 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.152802 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhm7s\" (UniqueName: \"kubernetes.io/projected/915ff449-5d83-4605-9ab8-e2ba528028a8-kube-api-access-vhm7s\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.153021 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.152959 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-data\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.153074 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.153034 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-workload-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.153126 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.153099 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-credential-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.153179 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.153164 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-envoy\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.153217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.153204 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-podinfo\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.153287 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.153270 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915ff449-5d83-4605-9ab8-e2ba528028a8-istiod-ca-cert\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.153355 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.153302 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-workload-certs\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.153398 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.153362 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-token\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.153436 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.153400 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-data\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.153436 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.153410 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-workload-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.153860 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.153831 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-workload-certs\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.153965 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.153883 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-credential-socket\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.154476 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.154447 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915ff449-5d83-4605-9ab8-e2ba528028a8-istiod-ca-cert\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.155947 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.155920 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-envoy\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.156292 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.156272 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-podinfo\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.181239 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.181204 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhm7s\" (UniqueName: \"kubernetes.io/projected/915ff449-5d83-4605-9ab8-e2ba528028a8-kube-api-access-vhm7s\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.182877 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.182853 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-token\") pod \"e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.298552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.298454 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:42.690229 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:34:42.690189 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915ff449_5d83_4605_9ab8_e2ba528028a8.slice/crio-b1ff9723e3bf4ae6b21d73fc26ea3872d40d225cdeae6307c27fe68030e6d3eb WatchSource:0}: Error finding container b1ff9723e3bf4ae6b21d73fc26ea3872d40d225cdeae6307c27fe68030e6d3eb: Status 404 returned error can't find the container with id b1ff9723e3bf4ae6b21d73fc26ea3872d40d225cdeae6307c27fe68030e6d3eb Oct 30 13:34:42.690909 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.690883 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9"] Oct 30 13:34:42.694145 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.694100 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:34:42.694257 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.694193 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:34:42.694315 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.694277 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:34:42.883508 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.882988 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" event={"ID":"915ff449-5d83-4605-9ab8-e2ba528028a8","Type":"ContainerStarted","Data":"30a989274d796ecb0574d2c6ae18fe8cf5b25052937ef787ba0b76a0bc938e7f"} Oct 30 13:34:42.883508 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.883061 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" event={"ID":"915ff449-5d83-4605-9ab8-e2ba528028a8","Type":"ContainerStarted","Data":"b1ff9723e3bf4ae6b21d73fc26ea3872d40d225cdeae6307c27fe68030e6d3eb"} Oct 30 13:34:42.937566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:42.937423 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" podStartSLOduration=1.937403193 podStartE2EDuration="1.937403193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:42.937164962 +0000 UTC m=+2539.569352182" watchObservedRunningTime="2026-10-30 13:34:42.937403193 +0000 UTC m=+2539.569590355" Oct 30 13:34:43.299282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:43.299242 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:43.301962 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:43.301925 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 13:34:43.302479 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:43.301973 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" podUID="915ff449-5d83-4605-9ab8-e2ba528028a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 13:34:43.875617 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:43.875570 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 13:34:43.875862 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:43.875628 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" podUID="1589354d-4858-4ad9-a76f-305adc4c83a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 13:34:43.926197 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:43.926162 7010 generic.go:356] "Generic (PLEG): container finished" podID="1589354d-4858-4ad9-a76f-305adc4c83a9" containerID="96eb82a94d4f48ae413198095b2aeede03c013b977864e0efff8d18c2d68dcf1" exitCode=0 Oct 30 13:34:43.926394 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:43.926245 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" event={"ID":"1589354d-4858-4ad9-a76f-305adc4c83a9","Type":"ContainerDied","Data":"96eb82a94d4f48ae413198095b2aeede03c013b977864e0efff8d18c2d68dcf1"} Oct 30 13:34:43.927037 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:43.927016 7010 scope.go:122] "RemoveContainer" containerID="96eb82a94d4f48ae413198095b2aeede03c013b977864e0efff8d18c2d68dcf1" Oct 30 13:34:43.929254 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:43.929218 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:34:43.929354 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:43.929315 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:34:43.929394 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:43.929360 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:34:44.300128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:44.300029 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 13:34:44.300789 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:44.300747 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" podUID="915ff449-5d83-4605-9ab8-e2ba528028a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 13:34:44.527465 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:44.527420 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf"] Oct 30 13:34:44.936739 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:44.936673 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" event={"ID":"1589354d-4858-4ad9-a76f-305adc4c83a9","Type":"ContainerStarted","Data":"8335bdfeb31c5f48284b621f49385bac23f9882d60406e5b21fa513f22315159"} Oct 30 13:34:45.299569 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:45.298911 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 13:34:45.299569 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:45.298974 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" podUID="915ff449-5d83-4605-9ab8-e2ba528028a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 13:34:45.849024 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:45.848947 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:34:45.849564 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:45.849081 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 13:34:45.849564 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:45.849125 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" podUID="1589354d-4858-4ad9-a76f-305adc4c83a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 13:34:46.300512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:46.299886 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 13:34:46.300512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:46.299947 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" podUID="915ff449-5d83-4605-9ab8-e2ba528028a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 13:34:46.844362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:46.844328 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 13:34:46.844616 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:46.844596 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" podUID="1589354d-4858-4ad9-a76f-305adc4c83a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 13:34:47.370293 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:47.370253 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 13:34:47.451627 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:47.448989 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" podUID="915ff449-5d83-4605-9ab8-e2ba528028a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 13:34:47.844564 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:47.844088 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 13:34:47.844564 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:47.844150 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" podUID="1589354d-4858-4ad9-a76f-305adc4c83a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 13:34:48.302514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:48.298938 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 13:34:48.302514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:48.299003 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" podUID="915ff449-5d83-4605-9ab8-e2ba528028a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 13:34:48.866511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:48.861699 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 13:34:48.866511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:48.861763 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" podUID="1589354d-4858-4ad9-a76f-305adc4c83a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 13:34:49.305021 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:49.304980 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 13:34:49.305313 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:49.305282 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" podUID="915ff449-5d83-4605-9ab8-e2ba528028a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 13:34:49.845953 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:49.845411 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 13:34:49.845953 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:49.845534 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" podUID="1589354d-4858-4ad9-a76f-305adc4c83a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 13:34:50.313516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:50.313466 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:50.314183 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:50.314157 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:50.314965 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:50.314948 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:34:50.580695 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:50.580603 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" podUID="1b3e9a3f-e471-4268-ac52-275511f40b7c" containerName="main" containerID="cri-o://1a3a0937377e03dc3d8f588590dd2f100dfd1ae2086623648546123745236e41" gracePeriod=45 Oct 30 13:34:50.847323 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:50.846865 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 13:34:50.847323 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:50.846926 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" podUID="1589354d-4858-4ad9-a76f-305adc4c83a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 13:34:50.999695 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:50.999666 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:34:51.136055 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.136011 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-model-cache\") pod \"1b3e9a3f-e471-4268-ac52-275511f40b7c\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " Oct 30 13:34:51.136254 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.136108 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3e9a3f-e471-4268-ac52-275511f40b7c-kube-api-access-h5q8w\" (UniqueName: \"kubernetes.io/projected/1b3e9a3f-e471-4268-ac52-275511f40b7c-kube-api-access-h5q8w\") pod \"1b3e9a3f-e471-4268-ac52-275511f40b7c\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " Oct 30 13:34:51.136330 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.136252 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-dshm\") pod \"1b3e9a3f-e471-4268-ac52-275511f40b7c\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " Oct 30 13:34:51.136330 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.136292 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b3e9a3f-e471-4268-ac52-275511f40b7c-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b3e9a3f-e471-4268-ac52-275511f40b7c-tls-certs\") pod \"1b3e9a3f-e471-4268-ac52-275511f40b7c\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " Oct 30 13:34:51.136434 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.136325 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-model-cache" pod "1b3e9a3f-e471-4268-ac52-275511f40b7c" (UID: "1b3e9a3f-e471-4268-ac52-275511f40b7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:51.136434 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.136386 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-tmp-dir\") pod \"1b3e9a3f-e471-4268-ac52-275511f40b7c\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " Oct 30 13:34:51.136434 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.136421 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-home\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-home\") pod \"1b3e9a3f-e471-4268-ac52-275511f40b7c\" (UID: \"1b3e9a3f-e471-4268-ac52-275511f40b7c\") " Oct 30 13:34:51.136728 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.136686 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-tmp-dir" pod "1b3e9a3f-e471-4268-ac52-275511f40b7c" (UID: "1b3e9a3f-e471-4268-ac52-275511f40b7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:51.136797 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.136755 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-home" pod "1b3e9a3f-e471-4268-ac52-275511f40b7c" (UID: "1b3e9a3f-e471-4268-ac52-275511f40b7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:51.137194 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.137168 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:34:51.137293 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.137199 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:34:51.137293 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.137213 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:34:51.138681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.138654 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b3e9a3f-e471-4268-ac52-275511f40b7c-tls-certs" pod "1b3e9a3f-e471-4268-ac52-275511f40b7c" (UID: "1b3e9a3f-e471-4268-ac52-275511f40b7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:34:51.138863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.138847 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3e9a3f-e471-4268-ac52-275511f40b7c-kube-api-access-h5q8w" pod "1b3e9a3f-e471-4268-ac52-275511f40b7c" (UID: "1b3e9a3f-e471-4268-ac52-275511f40b7c"). InnerVolumeSpecName "kube-api-access-h5q8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:51.138925 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.138875 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-dshm" pod "1b3e9a3f-e471-4268-ac52-275511f40b7c" (UID: "1b3e9a3f-e471-4268-ac52-275511f40b7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:51.187152 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.187115 7010 generic.go:356] "Generic (PLEG): container finished" podID="1b3e9a3f-e471-4268-ac52-275511f40b7c" containerID="1a3a0937377e03dc3d8f588590dd2f100dfd1ae2086623648546123745236e41" exitCode=0 Oct 30 13:34:51.187366 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.187184 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" Oct 30 13:34:51.187366 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.187200 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" event={"ID":"1b3e9a3f-e471-4268-ac52-275511f40b7c","Type":"ContainerDied","Data":"1a3a0937377e03dc3d8f588590dd2f100dfd1ae2086623648546123745236e41"} Oct 30 13:34:51.187366 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.187241 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" event={"ID":"1b3e9a3f-e471-4268-ac52-275511f40b7c","Type":"ContainerDied","Data":"b24a8f936bb2f6c6f23c7bb60ab3908396480ac2e995f0ed8905ad4445b5b01d"} Oct 30 13:34:51.187366 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.187258 7010 scope.go:122] "RemoveContainer" containerID="1a3a0937377e03dc3d8f588590dd2f100dfd1ae2086623648546123745236e41" Oct 30 13:34:51.199778 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.199754 7010 scope.go:122] "RemoveContainer" containerID="1a3a0937377e03dc3d8f588590dd2f100dfd1ae2086623648546123745236e41" Oct 30 13:34:51.200214 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:34:51.200186 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3a0937377e03dc3d8f588590dd2f100dfd1ae2086623648546123745236e41\": container with ID starting with 1a3a0937377e03dc3d8f588590dd2f100dfd1ae2086623648546123745236e41 not found: ID does not exist" containerID="1a3a0937377e03dc3d8f588590dd2f100dfd1ae2086623648546123745236e41" Oct 30 13:34:51.200283 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.200226 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3a0937377e03dc3d8f588590dd2f100dfd1ae2086623648546123745236e41"} err="failed to get container status \"1a3a0937377e03dc3d8f588590dd2f100dfd1ae2086623648546123745236e41\": rpc error: code = NotFound desc = could not find container \"1a3a0937377e03dc3d8f588590dd2f100dfd1ae2086623648546123745236e41\": container with ID starting with 1a3a0937377e03dc3d8f588590dd2f100dfd1ae2086623648546123745236e41 not found: ID does not exist" Oct 30 13:34:51.238880 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.238846 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5q8w\" (UniqueName: \"kubernetes.io/projected/1b3e9a3f-e471-4268-ac52-275511f40b7c-kube-api-access-h5q8w\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:34:51.238880 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.238879 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b3e9a3f-e471-4268-ac52-275511f40b7c-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:34:51.239145 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.238895 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b3e9a3f-e471-4268-ac52-275511f40b7c-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:34:51.243059 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.243016 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x"] Oct 30 13:34:51.246928 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.246901 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x"] Oct 30 13:34:51.489882 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.489802 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3e9a3f-e471-4268-ac52-275511f40b7c" path="/var/lib/kubelet/pods/1b3e9a3f-e471-4268-ac52-275511f40b7c/volumes" Oct 30 13:34:51.850428 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.850341 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:34:51.850768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.850736 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:34:51.851503 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.851476 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:34:51.913731 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:51.913694 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-c47903/auth-test-model-c47903-kserve-f5bbd9dc7-v2m7x" podUID="1b3e9a3f-e471-4268-ac52-275511f40b7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.129:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:34:59.404234 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:34:59.404188 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:34:59.555353 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:59.555313 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" podUID="95176be7-e786-4ddc-bf3f-99ae16ff9c20" containerName="main" containerID="cri-o://08a63d358d206f444c57f31d3075ff771b98e2d80265c63b34f1d77d12c40d51" gracePeriod=45 Oct 30 13:34:59.948894 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:59.948863 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:34:59.971868 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:59.971819 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-home\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-home\") pod \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " Oct 30 13:34:59.972046 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:59.971941 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-tmp-dir\") pod \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " Oct 30 13:34:59.972046 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:59.971979 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-dshm\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-dshm\") pod \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " Oct 30 13:34:59.972176 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:59.972053 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-model-cache\") pod \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " Oct 30 13:34:59.972176 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:59.972116 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-home" pod "95176be7-e786-4ddc-bf3f-99ae16ff9c20" (UID: "95176be7-e786-4ddc-bf3f-99ae16ff9c20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:59.972176 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:59.972125 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-tmp-dir" pod "95176be7-e786-4ddc-bf3f-99ae16ff9c20" (UID: "95176be7-e786-4ddc-bf3f-99ae16ff9c20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:59.972176 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:59.972156 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95176be7-e786-4ddc-bf3f-99ae16ff9c20-tls-certs\" (UniqueName: \"kubernetes.io/secret/95176be7-e786-4ddc-bf3f-99ae16ff9c20-tls-certs\") pod \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " Oct 30 13:34:59.972376 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:59.972214 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95176be7-e786-4ddc-bf3f-99ae16ff9c20-kube-api-access-q2722\" (UniqueName: \"kubernetes.io/projected/95176be7-e786-4ddc-bf3f-99ae16ff9c20-kube-api-access-q2722\") pod \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\" (UID: \"95176be7-e786-4ddc-bf3f-99ae16ff9c20\") " Oct 30 13:34:59.972376 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:59.972359 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-model-cache" pod "95176be7-e786-4ddc-bf3f-99ae16ff9c20" (UID: "95176be7-e786-4ddc-bf3f-99ae16ff9c20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:34:59.972825 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:59.972797 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:34:59.972934 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:59.972839 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:34:59.972934 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:59.972853 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:34:59.974418 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:59.974398 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95176be7-e786-4ddc-bf3f-99ae16ff9c20-kube-api-access-q2722" pod "95176be7-e786-4ddc-bf3f-99ae16ff9c20" (UID: "95176be7-e786-4ddc-bf3f-99ae16ff9c20"). InnerVolumeSpecName "kube-api-access-q2722". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:34:59.974669 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:59.974645 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95176be7-e786-4ddc-bf3f-99ae16ff9c20-tls-certs" pod "95176be7-e786-4ddc-bf3f-99ae16ff9c20" (UID: "95176be7-e786-4ddc-bf3f-99ae16ff9c20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:34:59.974764 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:34:59.974670 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-dshm" pod "95176be7-e786-4ddc-bf3f-99ae16ff9c20" (UID: "95176be7-e786-4ddc-bf3f-99ae16ff9c20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:00.096511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.095830 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95176be7-e786-4ddc-bf3f-99ae16ff9c20-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:35:00.096511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.095869 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95176be7-e786-4ddc-bf3f-99ae16ff9c20-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:35:00.096511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.095885 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2722\" (UniqueName: \"kubernetes.io/projected/95176be7-e786-4ddc-bf3f-99ae16ff9c20-kube-api-access-q2722\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:35:00.136132 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.136083 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km"] Oct 30 13:35:00.137037 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.137004 7010 cpu_manager.go:414] "containerMap: removing container" podUID="1b3e9a3f-e471-4268-ac52-275511f40b7c" containerName="main" Oct 30 13:35:00.137037 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.137034 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3e9a3f-e471-4268-ac52-275511f40b7c" containerName="main" Oct 30 13:35:00.137252 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.137107 7010 cpu_manager.go:414] "containerMap: removing container" podUID="95176be7-e786-4ddc-bf3f-99ae16ff9c20" containerName="main" Oct 30 13:35:00.137252 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.137113 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95176be7-e786-4ddc-bf3f-99ae16ff9c20" containerName="main" Oct 30 13:35:00.137252 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.137220 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="95176be7-e786-4ddc-bf3f-99ae16ff9c20" containerName="main" Oct 30 13:35:00.137252 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.137232 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3e9a3f-e471-4268-ac52-275511f40b7c" containerName="main" Oct 30 13:35:00.139836 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.139814 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" Oct 30 13:35:00.178583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.176763 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-082748\"" Oct 30 13:35:00.178583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.177391 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-082748-dockercfg-m6tv9\"" Oct 30 13:35:00.188610 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.181090 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km"] Oct 30 13:35:00.222513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.198761 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrg4n\" (UniqueName: \"kubernetes.io/projected/4252bbb1-b2b1-4e24-9867-7d93421825b5-kube-api-access-xrg4n\") pod \"payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km\" (UID: \"4252bbb1-b2b1-4e24-9867-7d93421825b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" Oct 30 13:35:00.260215 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.236145 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4252bbb1-b2b1-4e24-9867-7d93421825b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km\" (UID: \"4252bbb1-b2b1-4e24-9867-7d93421825b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" Oct 30 13:35:00.260215 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.236608 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc"] Oct 30 13:35:00.260215 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.241025 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" Oct 30 13:35:00.272157 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.272122 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc"] Oct 30 13:35:00.288308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.288272 7010 generic.go:356] "Generic (PLEG): container finished" podID="95176be7-e786-4ddc-bf3f-99ae16ff9c20" containerID="08a63d358d206f444c57f31d3075ff771b98e2d80265c63b34f1d77d12c40d51" exitCode=0 Oct 30 13:35:00.288608 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.288580 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" event={"ID":"95176be7-e786-4ddc-bf3f-99ae16ff9c20","Type":"ContainerDied","Data":"08a63d358d206f444c57f31d3075ff771b98e2d80265c63b34f1d77d12c40d51"} Oct 30 13:35:00.288862 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.288844 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" event={"ID":"95176be7-e786-4ddc-bf3f-99ae16ff9c20","Type":"ContainerDied","Data":"1328bf6e064f48c577043f52f8d216232e53164729fb6aef80061e00801c2cec"} Oct 30 13:35:00.289011 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.288792 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf" Oct 30 13:35:00.289984 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.288966 7010 scope.go:122] "RemoveContainer" containerID="08a63d358d206f444c57f31d3075ff771b98e2d80265c63b34f1d77d12c40d51" Oct 30 13:35:00.335929 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.335899 7010 scope.go:122] "RemoveContainer" containerID="08a63d358d206f444c57f31d3075ff771b98e2d80265c63b34f1d77d12c40d51" Oct 30 13:35:00.337129 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:35:00.336876 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a63d358d206f444c57f31d3075ff771b98e2d80265c63b34f1d77d12c40d51\": container with ID starting with 08a63d358d206f444c57f31d3075ff771b98e2d80265c63b34f1d77d12c40d51 not found: ID does not exist" containerID="08a63d358d206f444c57f31d3075ff771b98e2d80265c63b34f1d77d12c40d51" Oct 30 13:35:00.337129 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.336920 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a63d358d206f444c57f31d3075ff771b98e2d80265c63b34f1d77d12c40d51"} err="failed to get container status \"08a63d358d206f444c57f31d3075ff771b98e2d80265c63b34f1d77d12c40d51\": rpc error: code = NotFound desc = could not find container \"08a63d358d206f444c57f31d3075ff771b98e2d80265c63b34f1d77d12c40d51\": container with ID starting with 08a63d358d206f444c57f31d3075ff771b98e2d80265c63b34f1d77d12c40d51 not found: ID does not exist" Oct 30 13:35:00.338901 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.338872 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnkh8\" (UniqueName: \"kubernetes.io/projected/209c4a96-a64e-4251-86cb-80b61f95d020-kube-api-access-cnkh8\") pod \"payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc\" (UID: \"209c4a96-a64e-4251-86cb-80b61f95d020\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" Oct 30 13:35:00.339219 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.339196 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209c4a96-a64e-4251-86cb-80b61f95d020-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc\" (UID: \"209c4a96-a64e-4251-86cb-80b61f95d020\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" Oct 30 13:35:00.339561 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.339519 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrg4n\" (UniqueName: \"kubernetes.io/projected/4252bbb1-b2b1-4e24-9867-7d93421825b5-kube-api-access-xrg4n\") pod \"payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km\" (UID: \"4252bbb1-b2b1-4e24-9867-7d93421825b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" Oct 30 13:35:00.339860 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.339839 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4252bbb1-b2b1-4e24-9867-7d93421825b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km\" (UID: \"4252bbb1-b2b1-4e24-9867-7d93421825b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" Oct 30 13:35:00.340650 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.340628 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4252bbb1-b2b1-4e24-9867-7d93421825b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km\" (UID: \"4252bbb1-b2b1-4e24-9867-7d93421825b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" Oct 30 13:35:00.373551 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.366537 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf"] Oct 30 13:35:00.373551 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.371521 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b29df4/auth-test-model-b29df4-kserve-bffb9cccc-k7tvf"] Oct 30 13:35:00.373551 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.372668 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrg4n\" (UniqueName: \"kubernetes.io/projected/4252bbb1-b2b1-4e24-9867-7d93421825b5-kube-api-access-xrg4n\") pod \"payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km\" (UID: \"4252bbb1-b2b1-4e24-9867-7d93421825b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" Oct 30 13:35:00.441243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.441154 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209c4a96-a64e-4251-86cb-80b61f95d020-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc\" (UID: \"209c4a96-a64e-4251-86cb-80b61f95d020\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" Oct 30 13:35:00.441760 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.441433 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnkh8\" (UniqueName: \"kubernetes.io/projected/209c4a96-a64e-4251-86cb-80b61f95d020-kube-api-access-cnkh8\") pod \"payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc\" (UID: \"209c4a96-a64e-4251-86cb-80b61f95d020\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" Oct 30 13:35:00.442338 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.442314 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209c4a96-a64e-4251-86cb-80b61f95d020-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc\" (UID: \"209c4a96-a64e-4251-86cb-80b61f95d020\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" Oct 30 13:35:00.466765 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.466729 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnkh8\" (UniqueName: \"kubernetes.io/projected/209c4a96-a64e-4251-86cb-80b61f95d020-kube-api-access-cnkh8\") pod \"payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc\" (UID: \"209c4a96-a64e-4251-86cb-80b61f95d020\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" Oct 30 13:35:00.551901 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.551841 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" Oct 30 13:35:00.606458 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.606424 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" Oct 30 13:35:00.863983 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.863955 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km"] Oct 30 13:35:00.866066 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:35:00.866039 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4252bbb1_b2b1_4e24_9867_7d93421825b5.slice/crio-8475611a67f61751a158830d374ec147e50c34d84fdff5423a73dfc4017787e4 WatchSource:0}: Error finding container 8475611a67f61751a158830d374ec147e50c34d84fdff5423a73dfc4017787e4: Status 404 returned error can't find the container with id 8475611a67f61751a158830d374ec147e50c34d84fdff5423a73dfc4017787e4 Oct 30 13:35:00.868118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.868101 7010 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:35:00.943328 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:00.943284 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc"] Oct 30 13:35:00.944709 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:35:00.944682 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209c4a96_a64e_4251_86cb_80b61f95d020.slice/crio-c6f01f26569fcfb0beda9f81a61d32ddb9691af25b2914fceb9046c453889730 WatchSource:0}: Error finding container c6f01f26569fcfb0beda9f81a61d32ddb9691af25b2914fceb9046c453889730: Status 404 returned error can't find the container with id c6f01f26569fcfb0beda9f81a61d32ddb9691af25b2914fceb9046c453889730 Oct 30 13:35:01.298279 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:01.298183 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" event={"ID":"209c4a96-a64e-4251-86cb-80b61f95d020","Type":"ContainerStarted","Data":"c6f01f26569fcfb0beda9f81a61d32ddb9691af25b2914fceb9046c453889730"} Oct 30 13:35:01.299925 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:01.299888 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" event={"ID":"4252bbb1-b2b1-4e24-9867-7d93421825b5","Type":"ContainerStarted","Data":"8475611a67f61751a158830d374ec147e50c34d84fdff5423a73dfc4017787e4"} Oct 30 13:35:01.491577 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:01.491539 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95176be7-e786-4ddc-bf3f-99ae16ff9c20" path="/var/lib/kubelet/pods/95176be7-e786-4ddc-bf3f-99ae16ff9c20/volumes" Oct 30 13:35:03.322324 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:03.322220 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" event={"ID":"4252bbb1-b2b1-4e24-9867-7d93421825b5","Type":"ContainerStarted","Data":"a59a43cb5ab05c0d720af46ab658f4029e85fc67351f39202e81fe134cef9862"} Oct 30 13:35:03.322782 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:03.322316 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" Oct 30 13:35:03.324339 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:03.324308 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" event={"ID":"209c4a96-a64e-4251-86cb-80b61f95d020","Type":"ContainerStarted","Data":"104cc2e8425982347b4f51d56b55aa2ce46ff00ea5c16a437b3c06dc64a4d9e9"} Oct 30 13:35:03.324472 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:03.324385 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" Oct 30 13:35:03.368925 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:03.368851 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" podStartSLOduration=1.33577199 podStartE2EDuration="3.368830822s" totalImagesPullingTime="2.033058832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:03.364544666 +0000 UTC m=+2559.996731828" watchObservedRunningTime="2026-10-30 13:35:03.368830822 +0000 UTC m=+2560.001017984" Oct 30 13:35:03.418994 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:03.418924 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" podStartSLOduration=1.470714395 podStartE2EDuration="3.418901801s" totalImagesPullingTime="1.948187406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:03.416335528 +0000 UTC m=+2560.048522689" watchObservedRunningTime="2026-10-30 13:35:03.418901801 +0000 UTC m=+2560.051088965" Oct 30 13:35:14.334888 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:14.334857 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" Oct 30 13:35:14.335442 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:14.335044 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" Oct 30 13:35:23.133551 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:23.133449 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h"] Oct 30 13:35:23.139270 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:23.139247 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" Oct 30 13:35:23.158041 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:23.158005 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h"] Oct 30 13:35:23.171066 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:23.170925 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2b254d-dockercfg-jhskb\"" Oct 30 13:35:23.171066 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:23.171036 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2b254d\"" Oct 30 13:35:23.272108 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:23.272063 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/101a000c-04a7-482b-87a9-abee5668595a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h\" (UID: \"101a000c-04a7-482b-87a9-abee5668595a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" Oct 30 13:35:23.272310 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:23.272188 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdvsj\" (UniqueName: \"kubernetes.io/projected/101a000c-04a7-482b-87a9-abee5668595a-kube-api-access-fdvsj\") pod \"payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h\" (UID: \"101a000c-04a7-482b-87a9-abee5668595a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" Oct 30 13:35:23.376131 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:23.376091 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/101a000c-04a7-482b-87a9-abee5668595a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h\" (UID: \"101a000c-04a7-482b-87a9-abee5668595a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" Oct 30 13:35:23.376472 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:23.376448 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdvsj\" (UniqueName: \"kubernetes.io/projected/101a000c-04a7-482b-87a9-abee5668595a-kube-api-access-fdvsj\") pod \"payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h\" (UID: \"101a000c-04a7-482b-87a9-abee5668595a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" Oct 30 13:35:23.382740 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:23.377400 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/101a000c-04a7-482b-87a9-abee5668595a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h\" (UID: \"101a000c-04a7-482b-87a9-abee5668595a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" Oct 30 13:35:23.460524 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:23.460407 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdvsj\" (UniqueName: \"kubernetes.io/projected/101a000c-04a7-482b-87a9-abee5668595a-kube-api-access-fdvsj\") pod \"payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h\" (UID: \"101a000c-04a7-482b-87a9-abee5668595a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" Oct 30 13:35:23.512261 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:23.512212 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2b254d-dockercfg-jhskb\"" Oct 30 13:35:23.513432 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:23.513405 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" Oct 30 13:35:24.045817 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:24.045789 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h"] Oct 30 13:35:24.047264 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:35:24.047236 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101a000c_04a7_482b_87a9_abee5668595a.slice/crio-449553a90e34c372430743bfe8f3f102e627dba163c26a28c73c8e9fa42f1b55 WatchSource:0}: Error finding container 449553a90e34c372430743bfe8f3f102e627dba163c26a28c73c8e9fa42f1b55: Status 404 returned error can't find the container with id 449553a90e34c372430743bfe8f3f102e627dba163c26a28c73c8e9fa42f1b55 Oct 30 13:35:24.679100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:24.679054 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" event={"ID":"101a000c-04a7-482b-87a9-abee5668595a","Type":"ContainerStarted","Data":"0a72a914fb1405e8a9bba8020a0ddef4f93fe2d674aeea95fdbe5c902ebd318c"} Oct 30 13:35:24.679775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:24.679744 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" event={"ID":"101a000c-04a7-482b-87a9-abee5668595a","Type":"ContainerStarted","Data":"449553a90e34c372430743bfe8f3f102e627dba163c26a28c73c8e9fa42f1b55"} Oct 30 13:35:24.681100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:24.681081 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" Oct 30 13:35:24.752019 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:24.751960 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" podStartSLOduration=1.751945639 podStartE2EDuration="1.751945639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:24.727353436 +0000 UTC m=+2581.359540598" watchObservedRunningTime="2026-10-30 13:35:24.751945639 +0000 UTC m=+2581.384132800" Oct 30 13:35:24.910804 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:24.910767 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82"] Oct 30 13:35:24.915903 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:24.915877 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" Oct 30 13:35:24.936716 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:24.936636 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-cf3c79-dockercfg-mvmc7\"" Oct 30 13:35:24.938706 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:24.938683 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-cf3c79\"" Oct 30 13:35:24.946880 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:24.946846 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82"] Oct 30 13:35:25.075572 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:25.075531 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbsn8\" (UniqueName: \"kubernetes.io/projected/68f11fb1-4cae-4d71-ac24-162737ef7a60-kube-api-access-hbsn8\") pod \"payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82\" (UID: \"68f11fb1-4cae-4d71-ac24-162737ef7a60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" Oct 30 13:35:25.075938 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:25.075922 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f11fb1-4cae-4d71-ac24-162737ef7a60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82\" (UID: \"68f11fb1-4cae-4d71-ac24-162737ef7a60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" Oct 30 13:35:25.178951 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:25.178915 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f11fb1-4cae-4d71-ac24-162737ef7a60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82\" (UID: \"68f11fb1-4cae-4d71-ac24-162737ef7a60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" Oct 30 13:35:25.179147 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:25.179096 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbsn8\" (UniqueName: \"kubernetes.io/projected/68f11fb1-4cae-4d71-ac24-162737ef7a60-kube-api-access-hbsn8\") pod \"payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82\" (UID: \"68f11fb1-4cae-4d71-ac24-162737ef7a60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" Oct 30 13:35:25.179745 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:25.179712 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f11fb1-4cae-4d71-ac24-162737ef7a60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82\" (UID: \"68f11fb1-4cae-4d71-ac24-162737ef7a60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" Oct 30 13:35:25.207247 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:25.207154 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbsn8\" (UniqueName: \"kubernetes.io/projected/68f11fb1-4cae-4d71-ac24-162737ef7a60-kube-api-access-hbsn8\") pod \"payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82\" (UID: \"68f11fb1-4cae-4d71-ac24-162737ef7a60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" Oct 30 13:35:25.240666 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:25.240642 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" Oct 30 13:35:25.594978 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:35:25.594937 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f11fb1_4cae_4d71_ac24_162737ef7a60.slice/crio-3e962815f6b19b23fb996fc14cd880848b665adda10997693955ae3e826b34ee WatchSource:0}: Error finding container 3e962815f6b19b23fb996fc14cd880848b665adda10997693955ae3e826b34ee: Status 404 returned error can't find the container with id 3e962815f6b19b23fb996fc14cd880848b665adda10997693955ae3e826b34ee Oct 30 13:35:25.599216 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:25.599188 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82"] Oct 30 13:35:25.704576 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:25.704530 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" event={"ID":"68f11fb1-4cae-4d71-ac24-162737ef7a60","Type":"ContainerStarted","Data":"3e962815f6b19b23fb996fc14cd880848b665adda10997693955ae3e826b34ee"} Oct 30 13:35:26.712982 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:26.712933 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" event={"ID":"68f11fb1-4cae-4d71-ac24-162737ef7a60","Type":"ContainerStarted","Data":"48e898adf0c4ef7e873c3974121708500a65010e0a51489deb3195c420127c27"} Oct 30 13:35:26.713431 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:26.713233 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" Oct 30 13:35:26.758133 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:26.758089 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" podStartSLOduration=2.758074196 podStartE2EDuration="2.758074196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:26.755640598 +0000 UTC m=+2583.387827771" watchObservedRunningTime="2026-10-30 13:35:26.758074196 +0000 UTC m=+2583.390261357" Oct 30 13:35:29.403730 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:35:29.403624 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:35:36.715738 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:36.715701 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" Oct 30 13:35:37.722374 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:37.722347 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" Oct 30 13:35:47.647257 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:47.647222 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv"] Oct 30 13:35:47.647796 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:47.647447 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" podUID="8b9f8710-c8e9-4c17-ac76-0e511f952916" containerName="istio-proxy" containerID="cri-o://8ddd5e029357c181ffbd085859cd63cd77574ad8b7f50abd1e584880907d5cac" gracePeriod=30 Oct 30 13:35:48.603665 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.603609 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg"] Oct 30 13:35:48.609526 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.609504 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.616147 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.616120 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.616290 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.616156 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.616290 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.616179 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.616440 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.616409 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.616539 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.616450 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pkch\" (UniqueName: \"kubernetes.io/projected/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-kube-api-access-7pkch\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.616618 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.616578 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.616697 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.616680 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.616822 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.616715 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.616822 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.616737 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.631615 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.631583 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg"] Oct 30 13:35:48.717628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.717593 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.718076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.717641 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.718076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.717661 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.718076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.717751 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.718076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.717777 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.718076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.717795 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.718076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.717827 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.718076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.717848 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pkch\" (UniqueName: \"kubernetes.io/projected/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-kube-api-access-7pkch\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.718076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.717922 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.718562 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.718161 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-credential-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.718562 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.718222 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-workload-socket\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.718562 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.718368 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-workload-certs\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.718562 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.718408 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-data\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.718562 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.718445 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istiod-ca-cert\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.720223 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.720190 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-envoy\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.720360 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.720339 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-podinfo\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.746058 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.746026 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-token\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.746213 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.746179 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pkch\" (UniqueName: \"kubernetes.io/projected/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-kube-api-access-7pkch\") pod \"e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:48.924705 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:48.924671 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:49.259119 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:49.258324 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg"] Oct 30 13:35:49.268253 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:35:49.268202 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f15b5e8_0877_4f50_ae90_6bfaab9a5fb5.slice/crio-86e34be7bd0204f2576fef3c94727a515a23f42da949c2a685d518831cdf248d WatchSource:0}: Error finding container 86e34be7bd0204f2576fef3c94727a515a23f42da949c2a685d518831cdf248d: Status 404 returned error can't find the container with id 86e34be7bd0204f2576fef3c94727a515a23f42da949c2a685d518831cdf248d Oct 30 13:35:49.270961 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:49.270928 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:35:49.271026 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:49.270998 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:35:49.271063 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:49.271024 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:35:49.931389 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:49.931352 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" event={"ID":"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5","Type":"ContainerStarted","Data":"c0c43b907461ba520f28a6d93ab22d68da228512bac0dc97f5d2060e0cbcb5c4"} Oct 30 13:35:49.931903 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:49.931409 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" event={"ID":"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5","Type":"ContainerStarted","Data":"86e34be7bd0204f2576fef3c94727a515a23f42da949c2a685d518831cdf248d"} Oct 30 13:35:50.013511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:50.013153 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" podStartSLOduration=2.013133223 podStartE2EDuration="2.013133223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:49.99901697 +0000 UTC m=+2606.631204132" watchObservedRunningTime="2026-10-30 13:35:50.013133223 +0000 UTC m=+2606.645320387" Oct 30 13:35:50.924852 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:50.924819 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:50.926263 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:50.926241 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 13:35:50.926358 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:50.926286 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" podUID="5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 13:35:51.925247 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:51.925215 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 13:35:51.925700 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:51.925266 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" podUID="5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 13:35:52.925440 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:52.925359 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 13:35:52.925440 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:52.925419 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" podUID="5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 13:35:52.956356 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:52.956311 7010 generic.go:356] "Generic (PLEG): container finished" podID="8b9f8710-c8e9-4c17-ac76-0e511f952916" containerID="8ddd5e029357c181ffbd085859cd63cd77574ad8b7f50abd1e584880907d5cac" exitCode=0 Oct 30 13:35:52.956546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:52.956366 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" event={"ID":"8b9f8710-c8e9-4c17-ac76-0e511f952916","Type":"ContainerDied","Data":"8ddd5e029357c181ffbd085859cd63cd77574ad8b7f50abd1e584880907d5cac"} Oct 30 13:35:52.956546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:52.956423 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" event={"ID":"8b9f8710-c8e9-4c17-ac76-0e511f952916","Type":"ContainerDied","Data":"a25ccbb4515a48fcaf02de2660a4f2b1b8acf68770f824caa684887ba64175c6"} Oct 30 13:35:52.956546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:52.956446 7010 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a25ccbb4515a48fcaf02de2660a4f2b1b8acf68770f824caa684887ba64175c6" Oct 30 13:35:52.966340 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:52.966312 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:35:53.026873 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.026842 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-workload-certs\") pod \"8b9f8710-c8e9-4c17-ac76-0e511f952916\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " Oct 30 13:35:53.027071 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.026939 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9f8710-c8e9-4c17-ac76-0e511f952916-kube-api-access-jzpwd\" (UniqueName: \"kubernetes.io/projected/8b9f8710-c8e9-4c17-ac76-0e511f952916-kube-api-access-jzpwd\") pod \"8b9f8710-c8e9-4c17-ac76-0e511f952916\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " Oct 30 13:35:53.027071 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.027002 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-token\" (UniqueName: \"kubernetes.io/projected/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-token\") pod \"8b9f8710-c8e9-4c17-ac76-0e511f952916\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " Oct 30 13:35:53.027071 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.027045 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-podinfo\") pod \"8b9f8710-c8e9-4c17-ac76-0e511f952916\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " Oct 30 13:35:53.027237 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.027083 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-envoy\") pod \"8b9f8710-c8e9-4c17-ac76-0e511f952916\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " Oct 30 13:35:53.027237 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.027107 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-workload-certs" pod "8b9f8710-c8e9-4c17-ac76-0e511f952916" (UID: "8b9f8710-c8e9-4c17-ac76-0e511f952916"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:53.027237 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.027121 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b9f8710-c8e9-4c17-ac76-0e511f952916-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9f8710-c8e9-4c17-ac76-0e511f952916-istiod-ca-cert\") pod \"8b9f8710-c8e9-4c17-ac76-0e511f952916\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " Oct 30 13:35:53.027237 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.027216 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-credential-socket\") pod \"8b9f8710-c8e9-4c17-ac76-0e511f952916\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " Oct 30 13:35:53.027476 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.027449 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-workload-socket\") pod \"8b9f8710-c8e9-4c17-ac76-0e511f952916\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " Oct 30 13:35:53.027566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.027469 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-credential-socket" pod "8b9f8710-c8e9-4c17-ac76-0e511f952916" (UID: "8b9f8710-c8e9-4c17-ac76-0e511f952916"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:53.027832 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.027779 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9f8710-c8e9-4c17-ac76-0e511f952916-istiod-ca-cert" pod "8b9f8710-c8e9-4c17-ac76-0e511f952916" (UID: "8b9f8710-c8e9-4c17-ac76-0e511f952916"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:35:53.027832 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.027562 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-data\") pod \"8b9f8710-c8e9-4c17-ac76-0e511f952916\" (UID: \"8b9f8710-c8e9-4c17-ac76-0e511f952916\") " Oct 30 13:35:53.027996 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.027766 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-workload-socket" pod "8b9f8710-c8e9-4c17-ac76-0e511f952916" (UID: "8b9f8710-c8e9-4c17-ac76-0e511f952916"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:53.027996 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.027780 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-data" pod "8b9f8710-c8e9-4c17-ac76-0e511f952916" (UID: "8b9f8710-c8e9-4c17-ac76-0e511f952916"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:53.028630 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.028602 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.028743 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.028634 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.028743 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.028649 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.028743 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.028661 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9f8710-c8e9-4c17-ac76-0e511f952916-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.028743 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.028675 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.029660 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.029638 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-envoy" pod "8b9f8710-c8e9-4c17-ac76-0e511f952916" (UID: "8b9f8710-c8e9-4c17-ac76-0e511f952916"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:35:53.029772 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.029679 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-podinfo" pod "8b9f8710-c8e9-4c17-ac76-0e511f952916" (UID: "8b9f8710-c8e9-4c17-ac76-0e511f952916"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:35:53.029772 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.029697 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9f8710-c8e9-4c17-ac76-0e511f952916-kube-api-access-jzpwd" pod "8b9f8710-c8e9-4c17-ac76-0e511f952916" (UID: "8b9f8710-c8e9-4c17-ac76-0e511f952916"). InnerVolumeSpecName "kube-api-access-jzpwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:53.030198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.030172 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-token" pod "8b9f8710-c8e9-4c17-ac76-0e511f952916" (UID: "8b9f8710-c8e9-4c17-ac76-0e511f952916"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:35:53.130882 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.130846 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzpwd\" (UniqueName: \"kubernetes.io/projected/8b9f8710-c8e9-4c17-ac76-0e511f952916-kube-api-access-jzpwd\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.130882 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.130878 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.130882 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.130888 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.131203 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.130896 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9f8710-c8e9-4c17-ac76-0e511f952916-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:35:53.926130 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.926078 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 13:35:53.926597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.926159 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" podUID="5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 13:35:53.964389 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:53.964358 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv" Oct 30 13:35:54.057827 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:54.057788 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv"] Oct 30 13:35:54.071014 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:54.070973 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f0a31b-openshift-default-7d4c7b8489-4nbvv"] Oct 30 13:35:54.925592 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:54.925544 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 13:35:54.925776 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:54.925630 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" podUID="5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 13:35:55.490429 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:55.490396 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b9f8710-c8e9-4c17-ac76-0e511f952916" path="/var/lib/kubelet/pods/8b9f8710-c8e9-4c17-ac76-0e511f952916/volumes" Oct 30 13:35:55.930192 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:55.930160 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:55.930559 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:55.930530 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:55.931386 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:35:55.931367 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:35:59.402928 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:35:59.402881 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:36:02.000584 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.000548 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh"] Oct 30 13:36:02.001362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.001343 7010 cpu_manager.go:414] "containerMap: removing container" podUID="8b9f8710-c8e9-4c17-ac76-0e511f952916" containerName="istio-proxy" Oct 30 13:36:02.001412 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.001365 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9f8710-c8e9-4c17-ac76-0e511f952916" containerName="istio-proxy" Oct 30 13:36:02.001646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.001624 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9f8710-c8e9-4c17-ac76-0e511f952916" containerName="istio-proxy" Oct 30 13:36:02.007319 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.007294 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.029935 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.029902 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh"] Oct 30 13:36:02.038761 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.038733 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a8d3b8-openshift-default-dockercfg-95nbl\"" Oct 30 13:36:02.145338 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.145234 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-workload-certs\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.145691 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.145663 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istiod-ca-cert\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.145867 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.145849 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-podinfo\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.146152 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.146126 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-envoy\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.146311 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.146295 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-workload-socket\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.146411 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.146375 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-credential-socket\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.146526 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.146442 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-token\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.146526 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.146469 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6sv\" (UniqueName: \"kubernetes.io/projected/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-kube-api-access-fr6sv\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.146711 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.146678 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-data\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.251596 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.251501 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-envoy\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.251864 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.251847 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-workload-socket\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.251972 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.251955 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-credential-socket\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.252076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.252064 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-token\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.252209 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.252197 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6sv\" (UniqueName: \"kubernetes.io/projected/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-kube-api-access-fr6sv\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.252383 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.252369 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-data\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.252610 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.252595 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-workload-certs\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.252746 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.252730 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istiod-ca-cert\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.252845 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.252833 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-podinfo\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.260505 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.255844 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-workload-certs\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.260505 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.256426 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istiod-ca-cert\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.260505 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.256923 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-workload-socket\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.260505 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.257129 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-credential-socket\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.260505 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.257394 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-data\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.284577 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.268228 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-podinfo\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.286095 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.285759 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-envoy\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.297209 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.297136 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-token\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.301519 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.297435 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6sv\" (UniqueName: \"kubernetes.io/projected/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-kube-api-access-fr6sv\") pod \"e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.333338 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.332848 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:02.699717 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.699677 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh"] Oct 30 13:36:02.704540 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:36:02.704472 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d8d8e3_f2ca_45e5_bd85_7454ddac6d53.slice/crio-525ac58c56b3ff2e531797b8c6ef0c7db363ff2a9ddabb7646cd27ffa516f6df WatchSource:0}: Error finding container 525ac58c56b3ff2e531797b8c6ef0c7db363ff2a9ddabb7646cd27ffa516f6df: Status 404 returned error can't find the container with id 525ac58c56b3ff2e531797b8c6ef0c7db363ff2a9ddabb7646cd27ffa516f6df Oct 30 13:36:02.707408 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.707376 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:36:02.707506 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.707454 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:36:02.707554 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:02.707482 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:36:03.064430 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:03.064339 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" event={"ID":"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53","Type":"ContainerStarted","Data":"0d30e05091e905667d1a1994a64024b26cd2e7a944a7bb8ee82e87c8be805bc1"} Oct 30 13:36:03.064430 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:03.064403 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" event={"ID":"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53","Type":"ContainerStarted","Data":"525ac58c56b3ff2e531797b8c6ef0c7db363ff2a9ddabb7646cd27ffa516f6df"} Oct 30 13:36:03.111750 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:03.111683 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" podStartSLOduration=2.111665351 podStartE2EDuration="2.111665351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:03.11083659 +0000 UTC m=+2619.743023751" watchObservedRunningTime="2026-10-30 13:36:03.111665351 +0000 UTC m=+2619.743852525" Oct 30 13:36:03.334379 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:03.334286 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:03.340770 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:03.340741 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:04.072132 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:04.072040 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:04.073140 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:04.073120 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:36:07.663927 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:07.663886 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6"] Oct 30 13:36:07.713061 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:07.713023 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6"] Oct 30 13:36:07.713246 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:07.713154 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" Oct 30 13:36:07.731118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:07.731085 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a8d3b8\"" Oct 30 13:36:07.742310 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:07.742281 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a8d3b8-dockercfg-hlgm9\"" Oct 30 13:36:07.804921 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:07.804894 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10326e74-37cd-41d8-b783-a8d1aae34d8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6\" (UID: \"10326e74-37cd-41d8-b783-a8d1aae34d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" Oct 30 13:36:07.805067 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:07.804929 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd97q\" (UniqueName: \"kubernetes.io/projected/10326e74-37cd-41d8-b783-a8d1aae34d8e-kube-api-access-xd97q\") pod \"payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6\" (UID: \"10326e74-37cd-41d8-b783-a8d1aae34d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" Oct 30 13:36:07.906403 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:07.906374 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10326e74-37cd-41d8-b783-a8d1aae34d8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6\" (UID: \"10326e74-37cd-41d8-b783-a8d1aae34d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" Oct 30 13:36:07.906593 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:07.906407 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd97q\" (UniqueName: \"kubernetes.io/projected/10326e74-37cd-41d8-b783-a8d1aae34d8e-kube-api-access-xd97q\") pod \"payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6\" (UID: \"10326e74-37cd-41d8-b783-a8d1aae34d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" Oct 30 13:36:07.907143 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:07.907118 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10326e74-37cd-41d8-b783-a8d1aae34d8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6\" (UID: \"10326e74-37cd-41d8-b783-a8d1aae34d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" Oct 30 13:36:07.930628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:07.930572 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd97q\" (UniqueName: \"kubernetes.io/projected/10326e74-37cd-41d8-b783-a8d1aae34d8e-kube-api-access-xd97q\") pod \"payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6\" (UID: \"10326e74-37cd-41d8-b783-a8d1aae34d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" Oct 30 13:36:08.043670 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:08.043630 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" Oct 30 13:36:08.354908 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:08.354878 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6"] Oct 30 13:36:08.357395 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:36:08.357338 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10326e74_37cd_41d8_b783_a8d1aae34d8e.slice/crio-562e3bdaa26eacd8954423457c5ec788d4ae540331311ff85b43977581c96db3 WatchSource:0}: Error finding container 562e3bdaa26eacd8954423457c5ec788d4ae540331311ff85b43977581c96db3: Status 404 returned error can't find the container with id 562e3bdaa26eacd8954423457c5ec788d4ae540331311ff85b43977581c96db3 Oct 30 13:36:09.112362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:09.112319 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" event={"ID":"10326e74-37cd-41d8-b783-a8d1aae34d8e","Type":"ContainerStarted","Data":"5fb25c6ebd4ff8739243f3666237675260745a5727fd77341d8072a7bf865a2f"} Oct 30 13:36:09.112883 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:09.112383 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" event={"ID":"10326e74-37cd-41d8-b783-a8d1aae34d8e","Type":"ContainerStarted","Data":"562e3bdaa26eacd8954423457c5ec788d4ae540331311ff85b43977581c96db3"} Oct 30 13:36:09.112883 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:09.112685 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" Oct 30 13:36:09.158929 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:09.158877 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" podStartSLOduration=2.15886164 podStartE2EDuration="2.15886164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:09.157800889 +0000 UTC m=+2625.789988051" watchObservedRunningTime="2026-10-30 13:36:09.15886164 +0000 UTC m=+2625.791048801" Oct 30 13:36:10.235841 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.235807 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj"] Oct 30 13:36:10.240863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.240840 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:36:10.270350 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.270322 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-2b254d-serving-cert\"" Oct 30 13:36:10.274063 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.274036 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj"] Oct 30 13:36:10.362181 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.362146 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f34026b-12db-4043-a58f-6d0344549ae7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj\" (UID: \"5f34026b-12db-4043-a58f-6d0344549ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:36:10.362360 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.362205 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktnsz\" (UniqueName: \"kubernetes.io/projected/5f34026b-12db-4043-a58f-6d0344549ae7-kube-api-access-ktnsz\") pod \"maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj\" (UID: \"5f34026b-12db-4043-a58f-6d0344549ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:36:10.362360 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.362309 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f34026b-12db-4043-a58f-6d0344549ae7-metrics-certs\") pod \"maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj\" (UID: \"5f34026b-12db-4043-a58f-6d0344549ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:36:10.362442 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.362381 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f34026b-12db-4043-a58f-6d0344549ae7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj\" (UID: \"5f34026b-12db-4043-a58f-6d0344549ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:36:10.463321 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.463291 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f34026b-12db-4043-a58f-6d0344549ae7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj\" (UID: \"5f34026b-12db-4043-a58f-6d0344549ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:36:10.463457 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.463334 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktnsz\" (UniqueName: \"kubernetes.io/projected/5f34026b-12db-4043-a58f-6d0344549ae7-kube-api-access-ktnsz\") pod \"maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj\" (UID: \"5f34026b-12db-4043-a58f-6d0344549ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:36:10.463457 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.463373 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f34026b-12db-4043-a58f-6d0344549ae7-metrics-certs\") pod \"maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj\" (UID: \"5f34026b-12db-4043-a58f-6d0344549ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:36:10.463457 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.463398 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f34026b-12db-4043-a58f-6d0344549ae7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj\" (UID: \"5f34026b-12db-4043-a58f-6d0344549ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:36:10.464056 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.464029 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f34026b-12db-4043-a58f-6d0344549ae7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj\" (UID: \"5f34026b-12db-4043-a58f-6d0344549ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:36:10.465891 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.465871 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f34026b-12db-4043-a58f-6d0344549ae7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj\" (UID: \"5f34026b-12db-4043-a58f-6d0344549ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:36:10.466033 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.466013 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f34026b-12db-4043-a58f-6d0344549ae7-metrics-certs\") pod \"maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj\" (UID: \"5f34026b-12db-4043-a58f-6d0344549ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:36:10.508776 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.508679 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktnsz\" (UniqueName: \"kubernetes.io/projected/5f34026b-12db-4043-a58f-6d0344549ae7-kube-api-access-ktnsz\") pod \"maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj\" (UID: \"5f34026b-12db-4043-a58f-6d0344549ae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:36:10.555348 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.555314 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:36:10.874725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:10.874696 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj"] Oct 30 13:36:10.875948 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:36:10.875919 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f34026b_12db_4043_a58f_6d0344549ae7.slice/crio-d866a01ff74b7fe79c5c3bc4d55ef41317823641807857888c36cffd56b38502 WatchSource:0}: Error finding container d866a01ff74b7fe79c5c3bc4d55ef41317823641807857888c36cffd56b38502: Status 404 returned error can't find the container with id d866a01ff74b7fe79c5c3bc4d55ef41317823641807857888c36cffd56b38502 Oct 30 13:36:11.129996 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:11.129906 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" event={"ID":"5f34026b-12db-4043-a58f-6d0344549ae7","Type":"ContainerStarted","Data":"d866a01ff74b7fe79c5c3bc4d55ef41317823641807857888c36cffd56b38502"} Oct 30 13:36:12.138885 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:12.138848 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" event={"ID":"5f34026b-12db-4043-a58f-6d0344549ae7","Type":"ContainerStarted","Data":"7e1138c0afa7522f730c3f25e593181892c1c97b981d9ce65dee62f41a4bb060"} Oct 30 13:36:12.139375 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:12.138916 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:36:12.197536 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:12.197444 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" podStartSLOduration=1.7547592779999999 podStartE2EDuration="2.197430746s" totalImagesPullingTime="442.671468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:12.195596683 +0000 UTC m=+2628.827783845" watchObservedRunningTime="2026-10-30 13:36:12.197430746 +0000 UTC m=+2628.829617953" Oct 30 13:36:18.150464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:18.150435 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:36:20.122788 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:20.122759 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" Oct 30 13:36:29.402540 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:36:29.402500 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:36:31.899809 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:31.899776 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c"] Oct 30 13:36:31.905793 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:31.905769 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:31.927034 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:31.927008 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c"] Oct 30 13:36:31.936690 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:31.936659 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-82796d-openshift-default-dockercfg-rtjbw\"" Oct 30 13:36:31.938151 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:31.938123 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-data\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:31.938270 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:31.938166 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-podinfo\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:31.938597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:31.938296 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8103d2-d203-4fc6-9e85-e071908539ab-istiod-ca-cert\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:31.938597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:31.938335 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-envoy\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:31.938597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:31.938568 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-token\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:31.938778 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:31.938606 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-credential-socket\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:31.938836 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:31.938792 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-workload-certs\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:31.938890 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:31.938864 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-workload-socket\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:31.938955 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:31.938914 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nktmp\" (UniqueName: \"kubernetes.io/projected/2d8103d2-d203-4fc6-9e85-e071908539ab-kube-api-access-nktmp\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.040744 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.040708 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-token\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.040920 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.040812 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-credential-socket\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.040920 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.040905 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-workload-certs\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.041019 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.040968 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-workload-socket\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.041068 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.041021 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nktmp\" (UniqueName: \"kubernetes.io/projected/2d8103d2-d203-4fc6-9e85-e071908539ab-kube-api-access-nktmp\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.041106 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.041092 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-data\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.041163 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.041117 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-podinfo\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.041218 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.041184 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8103d2-d203-4fc6-9e85-e071908539ab-istiod-ca-cert\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.041276 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.041220 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-envoy\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.041331 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.041297 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-credential-socket\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.041331 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.041306 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-workload-certs\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.041734 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.041708 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-data\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.041899 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.041731 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-workload-socket\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.042030 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.041930 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8103d2-d203-4fc6-9e85-e071908539ab-istiod-ca-cert\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.043599 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.043577 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-envoy\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.043697 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.043678 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-podinfo\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.073394 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.073362 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-token\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.078431 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.078400 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nktmp\" (UniqueName: \"kubernetes.io/projected/2d8103d2-d203-4fc6-9e85-e071908539ab-kube-api-access-nktmp\") pod \"e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.220615 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.220536 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:32.561098 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.561075 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c"] Oct 30 13:36:32.562950 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:36:32.562923 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8103d2_d203_4fc6_9e85_e071908539ab.slice/crio-8514bc47a950251035678382ef40163c365fb16d77285ee0bdd5a020a474aa7d WatchSource:0}: Error finding container 8514bc47a950251035678382ef40163c365fb16d77285ee0bdd5a020a474aa7d: Status 404 returned error can't find the container with id 8514bc47a950251035678382ef40163c365fb16d77285ee0bdd5a020a474aa7d Oct 30 13:36:32.565771 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.565736 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:36:32.565858 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.565804 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:36:32.565900 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:32.565855 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:36:33.332437 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:33.332398 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" event={"ID":"2d8103d2-d203-4fc6-9e85-e071908539ab","Type":"ContainerStarted","Data":"e755a2c5e593690430564e91bdd7c88542082b17f964dfb454cca982903d63c1"} Oct 30 13:36:33.332865 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:33.332458 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" event={"ID":"2d8103d2-d203-4fc6-9e85-e071908539ab","Type":"ContainerStarted","Data":"8514bc47a950251035678382ef40163c365fb16d77285ee0bdd5a020a474aa7d"} Oct 30 13:36:33.380505 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:33.380447 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" podStartSLOduration=2.3804324230000002 podStartE2EDuration="2.380432423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:33.378353036 +0000 UTC m=+2650.010540198" watchObservedRunningTime="2026-10-30 13:36:33.380432423 +0000 UTC m=+2650.012619586" Oct 30 13:36:34.221753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:34.221719 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:34.223388 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:34.223361 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:36:34.223559 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:34.223415 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" podUID="2d8103d2-d203-4fc6-9e85-e071908539ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:36:35.221729 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:35.221691 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:36:35.222138 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:35.221749 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" podUID="2d8103d2-d203-4fc6-9e85-e071908539ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:36:36.221688 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:36.221647 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:36:36.221870 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:36.221707 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" podUID="2d8103d2-d203-4fc6-9e85-e071908539ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:36:37.221296 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:37.221260 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:36:37.221503 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:37.221339 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" podUID="2d8103d2-d203-4fc6-9e85-e071908539ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:36:38.221271 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:38.221223 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:36:38.221748 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:38.221311 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" podUID="2d8103d2-d203-4fc6-9e85-e071908539ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:36:39.221101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:39.221056 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 13:36:39.221324 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:39.221123 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" podUID="2d8103d2-d203-4fc6-9e85-e071908539ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 13:36:40.226349 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:40.226315 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:40.226779 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:40.226745 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:40.227599 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:40.227579 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:36:45.229281 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.229243 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz"] Oct 30 13:36:45.229760 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.229498 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" podUID="8fa8a642-6cd9-4f8c-889f-462088c95ecf" containerName="maas-api" containerID="cri-o://be4bd166b9247c938475a7d9cfe8dc7e3521321bd6301bccdb2aa52c00a5e63d" gracePeriod=30 Oct 30 13:36:45.371791 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.371753 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" podUID="8fa8a642-6cd9-4f8c-889f-462088c95ecf" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.80:8443/health\": dial tcp 10.134.0.80:8443: connect: connection refused" Oct 30 13:36:45.641239 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.640789 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:36:45.688561 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.684036 7010 generic.go:356] "Generic (PLEG): container finished" podID="8fa8a642-6cd9-4f8c-889f-462088c95ecf" containerID="be4bd166b9247c938475a7d9cfe8dc7e3521321bd6301bccdb2aa52c00a5e63d" exitCode=0 Oct 30 13:36:45.688561 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.684830 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" event={"ID":"8fa8a642-6cd9-4f8c-889f-462088c95ecf","Type":"ContainerDied","Data":"be4bd166b9247c938475a7d9cfe8dc7e3521321bd6301bccdb2aa52c00a5e63d"} Oct 30 13:36:45.688561 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.684903 7010 scope.go:122] "RemoveContainer" containerID="be4bd166b9247c938475a7d9cfe8dc7e3521321bd6301bccdb2aa52c00a5e63d" Oct 30 13:36:45.752249 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.750164 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa8a642-6cd9-4f8c-889f-462088c95ecf-kube-api-access-8skx6\" (UniqueName: \"kubernetes.io/projected/8fa8a642-6cd9-4f8c-889f-462088c95ecf-kube-api-access-8skx6\") pod \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\" (UID: \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\") " Oct 30 13:36:45.752249 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.750224 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa8a642-6cd9-4f8c-889f-462088c95ecf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa8a642-6cd9-4f8c-889f-462088c95ecf-openshift-service-ca\") pod \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\" (UID: \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\") " Oct 30 13:36:45.752249 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.750369 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa8a642-6cd9-4f8c-889f-462088c95ecf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa8a642-6cd9-4f8c-889f-462088c95ecf-maas-api-tls\") pod \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\" (UID: \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\") " Oct 30 13:36:45.752249 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.750469 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa8a642-6cd9-4f8c-889f-462088c95ecf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fa8a642-6cd9-4f8c-889f-462088c95ecf-metrics-certs\") pod \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\" (UID: \"8fa8a642-6cd9-4f8c-889f-462088c95ecf\") " Oct 30 13:36:45.788925 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.775814 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa8a642-6cd9-4f8c-889f-462088c95ecf-openshift-service-ca" pod "8fa8a642-6cd9-4f8c-889f-462088c95ecf" (UID: "8fa8a642-6cd9-4f8c-889f-462088c95ecf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:36:45.801113 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.799687 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa8a642-6cd9-4f8c-889f-462088c95ecf-metrics-certs" pod "8fa8a642-6cd9-4f8c-889f-462088c95ecf" (UID: "8fa8a642-6cd9-4f8c-889f-462088c95ecf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:36:45.867513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.860674 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa8a642-6cd9-4f8c-889f-462088c95ecf-kube-api-access-8skx6" pod "8fa8a642-6cd9-4f8c-889f-462088c95ecf" (UID: "8fa8a642-6cd9-4f8c-889f-462088c95ecf"). InnerVolumeSpecName "kube-api-access-8skx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:36:45.867513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.861281 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8skx6\" (UniqueName: \"kubernetes.io/projected/8fa8a642-6cd9-4f8c-889f-462088c95ecf-kube-api-access-8skx6\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:36:45.867513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.861308 7010 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa8a642-6cd9-4f8c-889f-462088c95ecf-openshift-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:36:45.867513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.861323 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fa8a642-6cd9-4f8c-889f-462088c95ecf-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:36:45.867513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.861471 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa8a642-6cd9-4f8c-889f-462088c95ecf-maas-api-tls" pod "8fa8a642-6cd9-4f8c-889f-462088c95ecf" (UID: "8fa8a642-6cd9-4f8c-889f-462088c95ecf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:36:45.963545 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:45.963509 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa8a642-6cd9-4f8c-889f-462088c95ecf-maas-api-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:36:46.692679 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:46.692631 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" event={"ID":"8fa8a642-6cd9-4f8c-889f-462088c95ecf","Type":"ContainerDied","Data":"3981adc34a0d6b99a0bdd6347f789aedddf0009f4797f75291f016b420e97c0b"} Oct 30 13:36:46.693125 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:46.692692 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz" Oct 30 13:36:46.749385 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:46.749344 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz"] Oct 30 13:36:46.761877 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:46.761843 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b29df4-86bb6fd959-fpbjz"] Oct 30 13:36:47.492968 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:47.492935 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa8a642-6cd9-4f8c-889f-462088c95ecf" path="/var/lib/kubelet/pods/8fa8a642-6cd9-4f8c-889f-462088c95ecf/volumes" Oct 30 13:36:59.343058 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:36:59.343010 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2"] Oct 30 13:36:59.408424 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:36:59.408387 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:37:07.984000 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:07.983950 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h"] Oct 30 13:37:12.985268 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:12.985195 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" podUID="101a000c-04a7-482b-87a9-abee5668595a" containerName="payload-pre-processing" containerID="cri-o://0a72a914fb1405e8a9bba8020a0ddef4f93fe2d674aeea95fdbe5c902ebd318c" gracePeriod=25 Oct 30 13:37:13.272860 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:13.272825 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" Oct 30 13:37:13.420484 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:13.420454 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101a000c-04a7-482b-87a9-abee5668595a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/101a000c-04a7-482b-87a9-abee5668595a-plugins-config-volume\") pod \"101a000c-04a7-482b-87a9-abee5668595a\" (UID: \"101a000c-04a7-482b-87a9-abee5668595a\") " Oct 30 13:37:13.420721 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:13.420702 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101a000c-04a7-482b-87a9-abee5668595a-kube-api-access-fdvsj\" (UniqueName: \"kubernetes.io/projected/101a000c-04a7-482b-87a9-abee5668595a-kube-api-access-fdvsj\") pod \"101a000c-04a7-482b-87a9-abee5668595a\" (UID: \"101a000c-04a7-482b-87a9-abee5668595a\") " Oct 30 13:37:13.420906 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:13.420885 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101a000c-04a7-482b-87a9-abee5668595a-plugins-config-volume" pod "101a000c-04a7-482b-87a9-abee5668595a" (UID: "101a000c-04a7-482b-87a9-abee5668595a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:37:13.421251 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:13.421234 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/101a000c-04a7-482b-87a9-abee5668595a-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:13.422810 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:13.422788 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101a000c-04a7-482b-87a9-abee5668595a-kube-api-access-fdvsj" pod "101a000c-04a7-482b-87a9-abee5668595a" (UID: "101a000c-04a7-482b-87a9-abee5668595a"). InnerVolumeSpecName "kube-api-access-fdvsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:13.523119 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:13.523083 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdvsj\" (UniqueName: \"kubernetes.io/projected/101a000c-04a7-482b-87a9-abee5668595a-kube-api-access-fdvsj\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:13.926009 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:13.925974 7010 generic.go:356] "Generic (PLEG): container finished" podID="101a000c-04a7-482b-87a9-abee5668595a" containerID="0a72a914fb1405e8a9bba8020a0ddef4f93fe2d674aeea95fdbe5c902ebd318c" exitCode=0 Oct 30 13:37:13.926217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:13.926018 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" Oct 30 13:37:13.926217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:13.926056 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" event={"ID":"101a000c-04a7-482b-87a9-abee5668595a","Type":"ContainerDied","Data":"0a72a914fb1405e8a9bba8020a0ddef4f93fe2d674aeea95fdbe5c902ebd318c"} Oct 30 13:37:13.926217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:13.926119 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h" event={"ID":"101a000c-04a7-482b-87a9-abee5668595a","Type":"ContainerDied","Data":"449553a90e34c372430743bfe8f3f102e627dba163c26a28c73c8e9fa42f1b55"} Oct 30 13:37:13.926217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:13.926151 7010 scope.go:122] "RemoveContainer" containerID="0a72a914fb1405e8a9bba8020a0ddef4f93fe2d674aeea95fdbe5c902ebd318c" Oct 30 13:37:13.937117 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:13.937089 7010 scope.go:122] "RemoveContainer" containerID="0a72a914fb1405e8a9bba8020a0ddef4f93fe2d674aeea95fdbe5c902ebd318c" Oct 30 13:37:13.937427 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:37:13.937405 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a72a914fb1405e8a9bba8020a0ddef4f93fe2d674aeea95fdbe5c902ebd318c\": container with ID starting with 0a72a914fb1405e8a9bba8020a0ddef4f93fe2d674aeea95fdbe5c902ebd318c not found: ID does not exist" containerID="0a72a914fb1405e8a9bba8020a0ddef4f93fe2d674aeea95fdbe5c902ebd318c" Oct 30 13:37:13.937507 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:13.937436 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a72a914fb1405e8a9bba8020a0ddef4f93fe2d674aeea95fdbe5c902ebd318c"} err="failed to get container status \"0a72a914fb1405e8a9bba8020a0ddef4f93fe2d674aeea95fdbe5c902ebd318c\": rpc error: code = NotFound desc = could not find container \"0a72a914fb1405e8a9bba8020a0ddef4f93fe2d674aeea95fdbe5c902ebd318c\": container with ID starting with 0a72a914fb1405e8a9bba8020a0ddef4f93fe2d674aeea95fdbe5c902ebd318c not found: ID does not exist" Oct 30 13:37:14.053559 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.053516 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h"] Oct 30 13:37:14.111559 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.111527 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-56d99db4b7-c262h"] Oct 30 13:37:14.338759 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.338664 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82"] Oct 30 13:37:14.371544 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.371474 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" containerID="cri-o://17dab731fefe765db2ad6b1ff9ec1ed23cf0aa488e5ce44c3af92548882e11d1" gracePeriod=45 Oct 30 13:37:14.716431 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.716403 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:37:14.775000 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.774918 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3264707d-9edc-4a4e-976f-c4d3aed728b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/3264707d-9edc-4a4e-976f-c4d3aed728b2-tls-certs\") pod \"3264707d-9edc-4a4e-976f-c4d3aed728b2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " Oct 30 13:37:14.775236 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.775100 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-model-cache\") pod \"3264707d-9edc-4a4e-976f-c4d3aed728b2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " Oct 30 13:37:14.775236 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.775162 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-tmp-dir\") pod \"3264707d-9edc-4a4e-976f-c4d3aed728b2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " Oct 30 13:37:14.775364 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.775236 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-dshm\") pod \"3264707d-9edc-4a4e-976f-c4d3aed728b2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " Oct 30 13:37:14.775364 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.775272 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3264707d-9edc-4a4e-976f-c4d3aed728b2-kube-api-access-l6m96\" (UniqueName: \"kubernetes.io/projected/3264707d-9edc-4a4e-976f-c4d3aed728b2-kube-api-access-l6m96\") pod \"3264707d-9edc-4a4e-976f-c4d3aed728b2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " Oct 30 13:37:14.775483 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.775384 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-home\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-home\") pod \"3264707d-9edc-4a4e-976f-c4d3aed728b2\" (UID: \"3264707d-9edc-4a4e-976f-c4d3aed728b2\") " Oct 30 13:37:14.775483 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.775386 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-model-cache" pod "3264707d-9edc-4a4e-976f-c4d3aed728b2" (UID: "3264707d-9edc-4a4e-976f-c4d3aed728b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:14.775837 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.775812 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-home" pod "3264707d-9edc-4a4e-976f-c4d3aed728b2" (UID: "3264707d-9edc-4a4e-976f-c4d3aed728b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:14.775938 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.775902 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-tmp-dir" pod "3264707d-9edc-4a4e-976f-c4d3aed728b2" (UID: "3264707d-9edc-4a4e-976f-c4d3aed728b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:14.776281 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.776258 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:14.776377 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.776287 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:14.776377 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.776302 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:14.778552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.778476 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-dshm" pod "3264707d-9edc-4a4e-976f-c4d3aed728b2" (UID: "3264707d-9edc-4a4e-976f-c4d3aed728b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:14.779673 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.779652 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3264707d-9edc-4a4e-976f-c4d3aed728b2-tls-certs" pod "3264707d-9edc-4a4e-976f-c4d3aed728b2" (UID: "3264707d-9edc-4a4e-976f-c4d3aed728b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:37:14.784370 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.783712 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3264707d-9edc-4a4e-976f-c4d3aed728b2-kube-api-access-l6m96" pod "3264707d-9edc-4a4e-976f-c4d3aed728b2" (UID: "3264707d-9edc-4a4e-976f-c4d3aed728b2"). InnerVolumeSpecName "kube-api-access-l6m96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:14.877315 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.877274 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3264707d-9edc-4a4e-976f-c4d3aed728b2-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:14.877315 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.877312 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6m96\" (UniqueName: \"kubernetes.io/projected/3264707d-9edc-4a4e-976f-c4d3aed728b2-kube-api-access-l6m96\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:14.877315 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:14.877322 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3264707d-9edc-4a4e-976f-c4d3aed728b2-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:15.064565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:15.060815 7010 generic.go:356] "Generic (PLEG): container finished" podID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerID="17dab731fefe765db2ad6b1ff9ec1ed23cf0aa488e5ce44c3af92548882e11d1" exitCode=0 Oct 30 13:37:15.064565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:15.060935 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" event={"ID":"3264707d-9edc-4a4e-976f-c4d3aed728b2","Type":"ContainerDied","Data":"17dab731fefe765db2ad6b1ff9ec1ed23cf0aa488e5ce44c3af92548882e11d1"} Oct 30 13:37:15.064565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:15.060972 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" event={"ID":"3264707d-9edc-4a4e-976f-c4d3aed728b2","Type":"ContainerDied","Data":"d202ccb3beb01ecfb0163144ed7446157bd7efe89ae8cfc0084180b916e1c890"} Oct 30 13:37:15.064565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:15.060997 7010 scope.go:122] "RemoveContainer" containerID="17dab731fefe765db2ad6b1ff9ec1ed23cf0aa488e5ce44c3af92548882e11d1" Oct 30 13:37:15.064565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:15.061151 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2" Oct 30 13:37:15.114729 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:15.114701 7010 scope.go:122] "RemoveContainer" containerID="17dab731fefe765db2ad6b1ff9ec1ed23cf0aa488e5ce44c3af92548882e11d1" Oct 30 13:37:15.115354 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:37:15.115329 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17dab731fefe765db2ad6b1ff9ec1ed23cf0aa488e5ce44c3af92548882e11d1\": container with ID starting with 17dab731fefe765db2ad6b1ff9ec1ed23cf0aa488e5ce44c3af92548882e11d1 not found: ID does not exist" containerID="17dab731fefe765db2ad6b1ff9ec1ed23cf0aa488e5ce44c3af92548882e11d1" Oct 30 13:37:15.116221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:15.116186 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17dab731fefe765db2ad6b1ff9ec1ed23cf0aa488e5ce44c3af92548882e11d1"} err="failed to get container status \"17dab731fefe765db2ad6b1ff9ec1ed23cf0aa488e5ce44c3af92548882e11d1\": rpc error: code = NotFound desc = could not find container \"17dab731fefe765db2ad6b1ff9ec1ed23cf0aa488e5ce44c3af92548882e11d1\": container with ID starting with 17dab731fefe765db2ad6b1ff9ec1ed23cf0aa488e5ce44c3af92548882e11d1 not found: ID does not exist" Oct 30 13:37:15.196892 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:15.196817 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2"] Oct 30 13:37:15.201804 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:15.201759 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d7e62b/simulator-w4-d7e62b-kserve-64cd54d9c7-6ppz2"] Oct 30 13:37:15.489733 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:15.489694 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101a000c-04a7-482b-87a9-abee5668595a" path="/var/lib/kubelet/pods/101a000c-04a7-482b-87a9-abee5668595a/volumes" Oct 30 13:37:15.490417 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:15.490398 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" path="/var/lib/kubelet/pods/3264707d-9edc-4a4e-976f-c4d3aed728b2/volumes" Oct 30 13:37:19.339257 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:19.339208 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" podUID="68f11fb1-4cae-4d71-ac24-162737ef7a60" containerName="payload-pre-processing" containerID="cri-o://48e898adf0c4ef7e873c3974121708500a65010e0a51489deb3195c420127c27" gracePeriod=25 Oct 30 13:37:19.655694 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:19.655667 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" Oct 30 13:37:19.793013 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:19.792972 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f11fb1-4cae-4d71-ac24-162737ef7a60-kube-api-access-hbsn8\" (UniqueName: \"kubernetes.io/projected/68f11fb1-4cae-4d71-ac24-162737ef7a60-kube-api-access-hbsn8\") pod \"68f11fb1-4cae-4d71-ac24-162737ef7a60\" (UID: \"68f11fb1-4cae-4d71-ac24-162737ef7a60\") " Oct 30 13:37:19.793226 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:19.793179 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f11fb1-4cae-4d71-ac24-162737ef7a60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f11fb1-4cae-4d71-ac24-162737ef7a60-plugins-config-volume\") pod \"68f11fb1-4cae-4d71-ac24-162737ef7a60\" (UID: \"68f11fb1-4cae-4d71-ac24-162737ef7a60\") " Oct 30 13:37:19.793594 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:19.793565 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f11fb1-4cae-4d71-ac24-162737ef7a60-plugins-config-volume" pod "68f11fb1-4cae-4d71-ac24-162737ef7a60" (UID: "68f11fb1-4cae-4d71-ac24-162737ef7a60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:37:19.793756 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:19.793739 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f11fb1-4cae-4d71-ac24-162737ef7a60-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:19.795107 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:19.795083 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f11fb1-4cae-4d71-ac24-162737ef7a60-kube-api-access-hbsn8" pod "68f11fb1-4cae-4d71-ac24-162737ef7a60" (UID: "68f11fb1-4cae-4d71-ac24-162737ef7a60"). InnerVolumeSpecName "kube-api-access-hbsn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:19.900994 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:19.899763 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbsn8\" (UniqueName: \"kubernetes.io/projected/68f11fb1-4cae-4d71-ac24-162737ef7a60-kube-api-access-hbsn8\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:20.125683 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:20.125645 7010 generic.go:356] "Generic (PLEG): container finished" podID="68f11fb1-4cae-4d71-ac24-162737ef7a60" containerID="48e898adf0c4ef7e873c3974121708500a65010e0a51489deb3195c420127c27" exitCode=0 Oct 30 13:37:20.125885 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:20.125711 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" Oct 30 13:37:20.125885 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:20.125764 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" event={"ID":"68f11fb1-4cae-4d71-ac24-162737ef7a60","Type":"ContainerDied","Data":"48e898adf0c4ef7e873c3974121708500a65010e0a51489deb3195c420127c27"} Oct 30 13:37:20.125885 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:20.125822 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82" event={"ID":"68f11fb1-4cae-4d71-ac24-162737ef7a60","Type":"ContainerDied","Data":"3e962815f6b19b23fb996fc14cd880848b665adda10997693955ae3e826b34ee"} Oct 30 13:37:20.125885 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:20.125854 7010 scope.go:122] "RemoveContainer" containerID="48e898adf0c4ef7e873c3974121708500a65010e0a51489deb3195c420127c27" Oct 30 13:37:20.137062 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:20.137043 7010 scope.go:122] "RemoveContainer" containerID="48e898adf0c4ef7e873c3974121708500a65010e0a51489deb3195c420127c27" Oct 30 13:37:20.137362 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:37:20.137343 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e898adf0c4ef7e873c3974121708500a65010e0a51489deb3195c420127c27\": container with ID starting with 48e898adf0c4ef7e873c3974121708500a65010e0a51489deb3195c420127c27 not found: ID does not exist" containerID="48e898adf0c4ef7e873c3974121708500a65010e0a51489deb3195c420127c27" Oct 30 13:37:20.137409 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:20.137379 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e898adf0c4ef7e873c3974121708500a65010e0a51489deb3195c420127c27"} err="failed to get container status \"48e898adf0c4ef7e873c3974121708500a65010e0a51489deb3195c420127c27\": rpc error: code = NotFound desc = could not find container \"48e898adf0c4ef7e873c3974121708500a65010e0a51489deb3195c420127c27\": container with ID starting with 48e898adf0c4ef7e873c3974121708500a65010e0a51489deb3195c420127c27 not found: ID does not exist" Oct 30 13:37:20.181646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:20.181553 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82"] Oct 30 13:37:20.184583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:20.184554 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-54fbd975cf-wpd82"] Oct 30 13:37:21.548177 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.548139 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f11fb1-4cae-4d71-ac24-162737ef7a60" path="/var/lib/kubelet/pods/68f11fb1-4cae-4d71-ac24-162737ef7a60/volumes" Oct 30 13:37:21.759859 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.759814 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp"] Oct 30 13:37:21.761310 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.761280 7010 cpu_manager.go:414] "containerMap: removing container" podUID="101a000c-04a7-482b-87a9-abee5668595a" containerName="payload-pre-processing" Oct 30 13:37:21.761558 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.761528 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101a000c-04a7-482b-87a9-abee5668595a" containerName="payload-pre-processing" Oct 30 13:37:21.761664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.761569 7010 cpu_manager.go:414] "containerMap: removing container" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" Oct 30 13:37:21.761664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.761577 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" Oct 30 13:37:21.761664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.761635 7010 cpu_manager.go:414] "containerMap: removing container" podUID="8fa8a642-6cd9-4f8c-889f-462088c95ecf" containerName="maas-api" Oct 30 13:37:21.761664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.761643 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa8a642-6cd9-4f8c-889f-462088c95ecf" containerName="maas-api" Oct 30 13:37:21.761873 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.761655 7010 cpu_manager.go:414] "containerMap: removing container" podUID="68f11fb1-4cae-4d71-ac24-162737ef7a60" containerName="payload-pre-processing" Oct 30 13:37:21.761873 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.761678 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f11fb1-4cae-4d71-ac24-162737ef7a60" containerName="payload-pre-processing" Oct 30 13:37:21.761993 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.761978 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa8a642-6cd9-4f8c-889f-462088c95ecf" containerName="maas-api" Oct 30 13:37:21.762107 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.762095 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f11fb1-4cae-4d71-ac24-162737ef7a60" containerName="payload-pre-processing" Oct 30 13:37:21.762209 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.762200 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="101a000c-04a7-482b-87a9-abee5668595a" containerName="payload-pre-processing" Oct 30 13:37:21.762301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.762291 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="3264707d-9edc-4a4e-976f-c4d3aed728b2" containerName="main" Oct 30 13:37:21.768620 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.768593 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" Oct 30 13:37:21.789525 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.786276 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp"] Oct 30 13:37:21.793782 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.793207 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-428a0e\"" Oct 30 13:37:21.793782 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.793333 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-428a0e-dockercfg-vc472\"" Oct 30 13:37:21.870614 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.870453 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069c8497-8a23-4175-b162-0e0adb63a7f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp\" (UID: \"069c8497-8a23-4175-b162-0e0adb63a7f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" Oct 30 13:37:21.870811 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.870699 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvnf4\" (UniqueName: \"kubernetes.io/projected/069c8497-8a23-4175-b162-0e0adb63a7f3-kube-api-access-xvnf4\") pod \"payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp\" (UID: \"069c8497-8a23-4175-b162-0e0adb63a7f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" Oct 30 13:37:21.897568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.897523 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr"] Oct 30 13:37:21.901995 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.901969 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" Oct 30 13:37:21.921964 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.921932 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr"] Oct 30 13:37:21.972299 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.972260 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069c8497-8a23-4175-b162-0e0adb63a7f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp\" (UID: \"069c8497-8a23-4175-b162-0e0adb63a7f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" Oct 30 13:37:21.972533 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.972364 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvnf4\" (UniqueName: \"kubernetes.io/projected/069c8497-8a23-4175-b162-0e0adb63a7f3-kube-api-access-xvnf4\") pod \"payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp\" (UID: \"069c8497-8a23-4175-b162-0e0adb63a7f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" Oct 30 13:37:21.973270 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.973242 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069c8497-8a23-4175-b162-0e0adb63a7f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp\" (UID: \"069c8497-8a23-4175-b162-0e0adb63a7f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" Oct 30 13:37:21.997402 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:21.997370 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvnf4\" (UniqueName: \"kubernetes.io/projected/069c8497-8a23-4175-b162-0e0adb63a7f3-kube-api-access-xvnf4\") pod \"payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp\" (UID: \"069c8497-8a23-4175-b162-0e0adb63a7f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" Oct 30 13:37:22.073567 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:22.073462 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwb8n\" (UniqueName: \"kubernetes.io/projected/e943dcc3-df3a-45aa-ab0c-07588840101b-kube-api-access-rwb8n\") pod \"payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr\" (UID: \"e943dcc3-df3a-45aa-ab0c-07588840101b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" Oct 30 13:37:22.073775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:22.073614 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e943dcc3-df3a-45aa-ab0c-07588840101b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr\" (UID: \"e943dcc3-df3a-45aa-ab0c-07588840101b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" Oct 30 13:37:22.103747 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:22.103706 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" Oct 30 13:37:22.176673 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:22.175745 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwb8n\" (UniqueName: \"kubernetes.io/projected/e943dcc3-df3a-45aa-ab0c-07588840101b-kube-api-access-rwb8n\") pod \"payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr\" (UID: \"e943dcc3-df3a-45aa-ab0c-07588840101b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" Oct 30 13:37:22.176673 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:22.175836 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e943dcc3-df3a-45aa-ab0c-07588840101b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr\" (UID: \"e943dcc3-df3a-45aa-ab0c-07588840101b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" Oct 30 13:37:22.176673 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:22.176623 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e943dcc3-df3a-45aa-ab0c-07588840101b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr\" (UID: \"e943dcc3-df3a-45aa-ab0c-07588840101b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" Oct 30 13:37:22.201282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:22.201247 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwb8n\" (UniqueName: \"kubernetes.io/projected/e943dcc3-df3a-45aa-ab0c-07588840101b-kube-api-access-rwb8n\") pod \"payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr\" (UID: \"e943dcc3-df3a-45aa-ab0c-07588840101b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" Oct 30 13:37:22.221293 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:22.221251 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" Oct 30 13:37:22.416953 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:22.416926 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp"] Oct 30 13:37:22.418883 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:37:22.418852 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069c8497_8a23_4175_b162_0e0adb63a7f3.slice/crio-861c14b6f48ea92afb7268c1953290d989b2916360dfa75bc3ae718c704aa64f WatchSource:0}: Error finding container 861c14b6f48ea92afb7268c1953290d989b2916360dfa75bc3ae718c704aa64f: Status 404 returned error can't find the container with id 861c14b6f48ea92afb7268c1953290d989b2916360dfa75bc3ae718c704aa64f Oct 30 13:37:22.545356 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:22.545322 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr"] Oct 30 13:37:22.546538 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:37:22.546510 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode943dcc3_df3a_45aa_ab0c_07588840101b.slice/crio-6c9d27a1f05159d5bf64d232cd80409533e4df1a190cd125a2de6e2552ab0c95 WatchSource:0}: Error finding container 6c9d27a1f05159d5bf64d232cd80409533e4df1a190cd125a2de6e2552ab0c95: Status 404 returned error can't find the container with id 6c9d27a1f05159d5bf64d232cd80409533e4df1a190cd125a2de6e2552ab0c95 Oct 30 13:37:23.152032 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:23.151998 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" event={"ID":"e943dcc3-df3a-45aa-ab0c-07588840101b","Type":"ContainerStarted","Data":"a07795828bfe325cf3916b06288c63960a7e8f7ac57724ca8051867104e95606"} Oct 30 13:37:23.152566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:23.152057 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" event={"ID":"e943dcc3-df3a-45aa-ab0c-07588840101b","Type":"ContainerStarted","Data":"6c9d27a1f05159d5bf64d232cd80409533e4df1a190cd125a2de6e2552ab0c95"} Oct 30 13:37:23.152566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:23.152082 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" Oct 30 13:37:23.153897 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:23.153872 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" event={"ID":"069c8497-8a23-4175-b162-0e0adb63a7f3","Type":"ContainerStarted","Data":"801c8aa3c21eb2ca562d66a810b309725d48620793aa53f0ab8ca86459e3b2c7"} Oct 30 13:37:23.154028 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:23.153911 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" event={"ID":"069c8497-8a23-4175-b162-0e0adb63a7f3","Type":"ContainerStarted","Data":"861c14b6f48ea92afb7268c1953290d989b2916360dfa75bc3ae718c704aa64f"} Oct 30 13:37:23.154028 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:23.153966 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" Oct 30 13:37:23.197898 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:23.197846 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" podStartSLOduration=2.197830028 podStartE2EDuration="2.197830028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:23.194047179 +0000 UTC m=+2699.826234341" watchObservedRunningTime="2026-10-30 13:37:23.197830028 +0000 UTC m=+2699.830017190" Oct 30 13:37:23.236862 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:23.236805 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" podStartSLOduration=2.236789421 podStartE2EDuration="2.236789421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:23.235731881 +0000 UTC m=+2699.867919042" watchObservedRunningTime="2026-10-30 13:37:23.236789421 +0000 UTC m=+2699.868976582" Oct 30 13:37:23.963393 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:23.963355 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:37:23.984758 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:23.984698 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:37:24.456733 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.456693 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj"] Oct 30 13:37:24.497514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.494852 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj"] Oct 30 13:37:24.497514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.494885 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" Oct 30 13:37:24.544527 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.541832 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68qq8\" (UniqueName: \"kubernetes.io/projected/d8dd1343-6d33-4c14-b861-0cc98485d047-kube-api-access-68qq8\") pod \"payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj\" (UID: \"d8dd1343-6d33-4c14-b861-0cc98485d047\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" Oct 30 13:37:24.544527 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.542052 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8dd1343-6d33-4c14-b861-0cc98485d047-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj\" (UID: \"d8dd1343-6d33-4c14-b861-0cc98485d047\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" Oct 30 13:37:24.582539 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.581752 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b4fb25\"" Oct 30 13:37:24.582911 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.582874 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b4fb25-dockercfg-r7k56\"" Oct 30 13:37:24.584927 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.584895 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c"] Oct 30 13:37:24.593183 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.593153 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" Oct 30 13:37:24.613774 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.613656 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c"] Oct 30 13:37:24.643636 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.643587 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8dd1343-6d33-4c14-b861-0cc98485d047-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj\" (UID: \"d8dd1343-6d33-4c14-b861-0cc98485d047\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" Oct 30 13:37:24.643899 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.643873 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68qq8\" (UniqueName: \"kubernetes.io/projected/d8dd1343-6d33-4c14-b861-0cc98485d047-kube-api-access-68qq8\") pod \"payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj\" (UID: \"d8dd1343-6d33-4c14-b861-0cc98485d047\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" Oct 30 13:37:24.644728 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.644704 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8dd1343-6d33-4c14-b861-0cc98485d047-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj\" (UID: \"d8dd1343-6d33-4c14-b861-0cc98485d047\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" Oct 30 13:37:24.670071 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.670036 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68qq8\" (UniqueName: \"kubernetes.io/projected/d8dd1343-6d33-4c14-b861-0cc98485d047-kube-api-access-68qq8\") pod \"payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj\" (UID: \"d8dd1343-6d33-4c14-b861-0cc98485d047\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" Oct 30 13:37:24.746025 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.745922 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5ec0d2-744a-4cff-a863-4aae9666eccc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c\" (UID: \"ca5ec0d2-744a-4cff-a863-4aae9666eccc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" Oct 30 13:37:24.746196 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.746046 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8szp\" (UniqueName: \"kubernetes.io/projected/ca5ec0d2-744a-4cff-a863-4aae9666eccc-kube-api-access-z8szp\") pod \"payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c\" (UID: \"ca5ec0d2-744a-4cff-a863-4aae9666eccc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" Oct 30 13:37:24.847998 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.847955 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5ec0d2-744a-4cff-a863-4aae9666eccc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c\" (UID: \"ca5ec0d2-744a-4cff-a863-4aae9666eccc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" Oct 30 13:37:24.848229 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.848043 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8szp\" (UniqueName: \"kubernetes.io/projected/ca5ec0d2-744a-4cff-a863-4aae9666eccc-kube-api-access-z8szp\") pod \"payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c\" (UID: \"ca5ec0d2-744a-4cff-a863-4aae9666eccc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" Oct 30 13:37:24.848699 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.848677 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5ec0d2-744a-4cff-a863-4aae9666eccc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c\" (UID: \"ca5ec0d2-744a-4cff-a863-4aae9666eccc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" Oct 30 13:37:24.873059 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.873021 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8szp\" (UniqueName: \"kubernetes.io/projected/ca5ec0d2-744a-4cff-a863-4aae9666eccc-kube-api-access-z8szp\") pod \"payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c\" (UID: \"ca5ec0d2-744a-4cff-a863-4aae9666eccc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" Oct 30 13:37:24.891967 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.891932 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" Oct 30 13:37:24.921870 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:24.921838 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" Oct 30 13:37:25.210880 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:25.210849 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj"] Oct 30 13:37:25.211561 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:37:25.211529 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8dd1343_6d33_4c14_b861_0cc98485d047.slice/crio-ba3820284acfd3f97426a03c8f9e74def41c49805fccc20c20df83e56f55775d WatchSource:0}: Error finding container ba3820284acfd3f97426a03c8f9e74def41c49805fccc20c20df83e56f55775d: Status 404 returned error can't find the container with id ba3820284acfd3f97426a03c8f9e74def41c49805fccc20c20df83e56f55775d Oct 30 13:37:25.267740 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:25.267715 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c"] Oct 30 13:37:25.274816 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:37:25.274783 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5ec0d2_744a_4cff_a863_4aae9666eccc.slice/crio-e2d36868a6c4272806958e5b84ff384e69aaa851b1b5a212a7ad1755bd4a6e1e WatchSource:0}: Error finding container e2d36868a6c4272806958e5b84ff384e69aaa851b1b5a212a7ad1755bd4a6e1e: Status 404 returned error can't find the container with id e2d36868a6c4272806958e5b84ff384e69aaa851b1b5a212a7ad1755bd4a6e1e Oct 30 13:37:26.226676 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:26.226629 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" event={"ID":"d8dd1343-6d33-4c14-b861-0cc98485d047","Type":"ContainerStarted","Data":"bb93944c9306f7fc2196b03772b249c7594ff48caa8ae987bf7c8bc2397f2677"} Oct 30 13:37:26.227154 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:26.226730 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" Oct 30 13:37:26.227154 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:26.226754 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" event={"ID":"d8dd1343-6d33-4c14-b861-0cc98485d047","Type":"ContainerStarted","Data":"ba3820284acfd3f97426a03c8f9e74def41c49805fccc20c20df83e56f55775d"} Oct 30 13:37:26.228245 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:26.228218 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" event={"ID":"ca5ec0d2-744a-4cff-a863-4aae9666eccc","Type":"ContainerStarted","Data":"c922d61609f3bfc2fb0d9ceefdb1689d93ad2f8f3e2163c0330041da09183c10"} Oct 30 13:37:26.228334 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:26.228263 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" event={"ID":"ca5ec0d2-744a-4cff-a863-4aae9666eccc","Type":"ContainerStarted","Data":"e2d36868a6c4272806958e5b84ff384e69aaa851b1b5a212a7ad1755bd4a6e1e"} Oct 30 13:37:26.228334 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:26.228283 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" Oct 30 13:37:26.271463 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:26.271410 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" podStartSLOduration=2.271396411 podStartE2EDuration="2.271396411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:26.269150865 +0000 UTC m=+2702.901338054" watchObservedRunningTime="2026-10-30 13:37:26.271396411 +0000 UTC m=+2702.903583573" Oct 30 13:37:26.311719 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:26.311656 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" podStartSLOduration=2.311638387 podStartE2EDuration="2.311638387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:26.311259347 +0000 UTC m=+2702.943446526" watchObservedRunningTime="2026-10-30 13:37:26.311638387 +0000 UTC m=+2702.943825550" Oct 30 13:37:29.401095 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:37:29.401056 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:37:34.213694 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:34.213600 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" Oct 30 13:37:34.214529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:34.214187 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" Oct 30 13:37:37.237634 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:37.237595 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" Oct 30 13:37:37.238130 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:37.237696 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" Oct 30 13:37:52.547026 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:52.546985 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59"] Oct 30 13:37:52.547529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:52.547199 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" podUID="b86738af-6a6b-428d-8141-c3c4eb82a923" containerName="istio-proxy" containerID="cri-o://aaf4b2a92fb636059b23c62319b41623a2b4f67f8fff84b1baffefe4e3e62f66" gracePeriod=30 Oct 30 13:37:57.844554 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.844503 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:37:57.871100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.871064 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-workload-socket\") pod \"b86738af-6a6b-428d-8141-c3c4eb82a923\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " Oct 30 13:37:57.871270 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.871174 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-podinfo\") pod \"b86738af-6a6b-428d-8141-c3c4eb82a923\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " Oct 30 13:37:57.871344 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.871328 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-data\") pod \"b86738af-6a6b-428d-8141-c3c4eb82a923\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " Oct 30 13:37:57.871409 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.871363 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-envoy\") pod \"b86738af-6a6b-428d-8141-c3c4eb82a923\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " Oct 30 13:37:57.871409 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.871369 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-workload-socket" pod "b86738af-6a6b-428d-8141-c3c4eb82a923" (UID: "b86738af-6a6b-428d-8141-c3c4eb82a923"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:57.871409 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.871402 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-credential-socket\") pod \"b86738af-6a6b-428d-8141-c3c4eb82a923\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " Oct 30 13:37:57.871628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.871432 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-token\" (UniqueName: \"kubernetes.io/projected/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-token\") pod \"b86738af-6a6b-428d-8141-c3c4eb82a923\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " Oct 30 13:37:57.871628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.871510 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86738af-6a6b-428d-8141-c3c4eb82a923-kube-api-access-rmpzp\" (UniqueName: \"kubernetes.io/projected/b86738af-6a6b-428d-8141-c3c4eb82a923-kube-api-access-rmpzp\") pod \"b86738af-6a6b-428d-8141-c3c4eb82a923\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " Oct 30 13:37:57.871628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.871570 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-workload-certs\") pod \"b86738af-6a6b-428d-8141-c3c4eb82a923\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " Oct 30 13:37:57.871798 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.871654 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b86738af-6a6b-428d-8141-c3c4eb82a923-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86738af-6a6b-428d-8141-c3c4eb82a923-istiod-ca-cert\") pod \"b86738af-6a6b-428d-8141-c3c4eb82a923\" (UID: \"b86738af-6a6b-428d-8141-c3c4eb82a923\") " Oct 30 13:37:57.871798 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.871657 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-data" pod "b86738af-6a6b-428d-8141-c3c4eb82a923" (UID: "b86738af-6a6b-428d-8141-c3c4eb82a923"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:57.871798 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.871717 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-credential-socket" pod "b86738af-6a6b-428d-8141-c3c4eb82a923" (UID: "b86738af-6a6b-428d-8141-c3c4eb82a923"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:57.872054 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.872031 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86738af-6a6b-428d-8141-c3c4eb82a923-istiod-ca-cert" pod "b86738af-6a6b-428d-8141-c3c4eb82a923" (UID: "b86738af-6a6b-428d-8141-c3c4eb82a923"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:37:57.872291 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.872262 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-workload-certs" pod "b86738af-6a6b-428d-8141-c3c4eb82a923" (UID: "b86738af-6a6b-428d-8141-c3c4eb82a923"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:57.873544 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.872509 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86738af-6a6b-428d-8141-c3c4eb82a923-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:57.873544 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.872535 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:57.873544 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.872550 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:57.873544 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.872565 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:57.873544 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.872580 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:57.873984 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.873959 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-envoy" pod "b86738af-6a6b-428d-8141-c3c4eb82a923" (UID: "b86738af-6a6b-428d-8141-c3c4eb82a923"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:37:57.874443 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.874408 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86738af-6a6b-428d-8141-c3c4eb82a923-kube-api-access-rmpzp" pod "b86738af-6a6b-428d-8141-c3c4eb82a923" (UID: "b86738af-6a6b-428d-8141-c3c4eb82a923"). InnerVolumeSpecName "kube-api-access-rmpzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:57.874590 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.874561 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-podinfo" pod "b86738af-6a6b-428d-8141-c3c4eb82a923" (UID: "b86738af-6a6b-428d-8141-c3c4eb82a923"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:37:57.875835 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.875804 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-token" pod "b86738af-6a6b-428d-8141-c3c4eb82a923" (UID: "b86738af-6a6b-428d-8141-c3c4eb82a923"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:37:57.974268 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.974230 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:57.974268 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.974261 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:57.974268 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.974272 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmpzp\" (UniqueName: \"kubernetes.io/projected/b86738af-6a6b-428d-8141-c3c4eb82a923-kube-api-access-rmpzp\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:57.974593 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:57.974282 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86738af-6a6b-428d-8141-c3c4eb82a923-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:37:58.169866 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:58.169718 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6"] Oct 30 13:37:58.541355 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:58.541269 7010 generic.go:356] "Generic (PLEG): container finished" podID="b86738af-6a6b-428d-8141-c3c4eb82a923" containerID="aaf4b2a92fb636059b23c62319b41623a2b4f67f8fff84b1baffefe4e3e62f66" exitCode=0 Oct 30 13:37:58.541355 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:58.541323 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" Oct 30 13:37:58.541653 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:58.541354 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" event={"ID":"b86738af-6a6b-428d-8141-c3c4eb82a923","Type":"ContainerDied","Data":"aaf4b2a92fb636059b23c62319b41623a2b4f67f8fff84b1baffefe4e3e62f66"} Oct 30 13:37:58.541653 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:58.541414 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59" event={"ID":"b86738af-6a6b-428d-8141-c3c4eb82a923","Type":"ContainerDied","Data":"df9cd4bf51844c119bdad51117e323563e8fcf679ac517ef4083535bb6299860"} Oct 30 13:37:58.541653 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:58.541450 7010 scope.go:122] "RemoveContainer" containerID="aaf4b2a92fb636059b23c62319b41623a2b4f67f8fff84b1baffefe4e3e62f66" Oct 30 13:37:58.554949 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:58.554925 7010 scope.go:122] "RemoveContainer" containerID="aaf4b2a92fb636059b23c62319b41623a2b4f67f8fff84b1baffefe4e3e62f66" Oct 30 13:37:58.555377 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:37:58.555349 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaf4b2a92fb636059b23c62319b41623a2b4f67f8fff84b1baffefe4e3e62f66\": container with ID starting with aaf4b2a92fb636059b23c62319b41623a2b4f67f8fff84b1baffefe4e3e62f66 not found: ID does not exist" containerID="aaf4b2a92fb636059b23c62319b41623a2b4f67f8fff84b1baffefe4e3e62f66" Oct 30 13:37:58.555447 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:58.555388 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaf4b2a92fb636059b23c62319b41623a2b4f67f8fff84b1baffefe4e3e62f66"} err="failed to get container status \"aaf4b2a92fb636059b23c62319b41623a2b4f67f8fff84b1baffefe4e3e62f66\": rpc error: code = NotFound desc = could not find container \"aaf4b2a92fb636059b23c62319b41623a2b4f67f8fff84b1baffefe4e3e62f66\": container with ID starting with aaf4b2a92fb636059b23c62319b41623a2b4f67f8fff84b1baffefe4e3e62f66 not found: ID does not exist" Oct 30 13:37:58.596978 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:58.596879 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59"] Oct 30 13:37:58.608579 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:58.608544 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5a6281-openshift-default-65f964f447-6fw59"] Oct 30 13:37:59.404462 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:37:59.404428 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:37:59.490614 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:37:59.490579 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b86738af-6a6b-428d-8141-c3c4eb82a923" path="/var/lib/kubelet/pods/b86738af-6a6b-428d-8141-c3c4eb82a923/volumes" Oct 30 13:38:00.489158 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.489116 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l"] Oct 30 13:38:00.490319 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.490296 7010 cpu_manager.go:414] "containerMap: removing container" podUID="b86738af-6a6b-428d-8141-c3c4eb82a923" containerName="istio-proxy" Oct 30 13:38:00.490434 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.490324 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86738af-6a6b-428d-8141-c3c4eb82a923" containerName="istio-proxy" Oct 30 13:38:00.490546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.490530 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86738af-6a6b-428d-8141-c3c4eb82a923" containerName="istio-proxy" Oct 30 13:38:00.499945 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.499915 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" Oct 30 13:38:00.564515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.555242 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2b254d\"" Oct 30 13:38:00.564515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.556075 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2b254d-dockercfg-fzr47\"" Oct 30 13:38:00.564515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.560392 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l"] Oct 30 13:38:00.574549 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.568556 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4pc5\" (UniqueName: \"kubernetes.io/projected/75a04bf8-d29c-425c-b351-61f785e861a3-kube-api-access-s4pc5\") pod \"payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l\" (UID: \"75a04bf8-d29c-425c-b351-61f785e861a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" Oct 30 13:38:00.574549 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.568908 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a04bf8-d29c-425c-b351-61f785e861a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l\" (UID: \"75a04bf8-d29c-425c-b351-61f785e861a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" Oct 30 13:38:00.675548 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.674514 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4pc5\" (UniqueName: \"kubernetes.io/projected/75a04bf8-d29c-425c-b351-61f785e861a3-kube-api-access-s4pc5\") pod \"payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l\" (UID: \"75a04bf8-d29c-425c-b351-61f785e861a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" Oct 30 13:38:00.675548 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.674724 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a04bf8-d29c-425c-b351-61f785e861a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l\" (UID: \"75a04bf8-d29c-425c-b351-61f785e861a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" Oct 30 13:38:00.675796 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.675749 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a04bf8-d29c-425c-b351-61f785e861a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l\" (UID: \"75a04bf8-d29c-425c-b351-61f785e861a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" Oct 30 13:38:00.706557 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.706472 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw"] Oct 30 13:38:00.714431 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.714366 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4pc5\" (UniqueName: \"kubernetes.io/projected/75a04bf8-d29c-425c-b351-61f785e861a3-kube-api-access-s4pc5\") pod \"payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l\" (UID: \"75a04bf8-d29c-425c-b351-61f785e861a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" Oct 30 13:38:00.723027 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.722992 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw"] Oct 30 13:38:00.723212 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.723064 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" Oct 30 13:38:00.776672 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.776627 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h5zz\" (UniqueName: \"kubernetes.io/projected/75756d4a-6197-4dfc-9efb-80cc78fc958d-kube-api-access-4h5zz\") pod \"payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw\" (UID: \"75756d4a-6197-4dfc-9efb-80cc78fc958d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" Oct 30 13:38:00.777089 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.777038 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75756d4a-6197-4dfc-9efb-80cc78fc958d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw\" (UID: \"75756d4a-6197-4dfc-9efb-80cc78fc958d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" Oct 30 13:38:00.878610 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.878571 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75756d4a-6197-4dfc-9efb-80cc78fc958d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw\" (UID: \"75756d4a-6197-4dfc-9efb-80cc78fc958d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" Oct 30 13:38:00.878858 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.878687 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h5zz\" (UniqueName: \"kubernetes.io/projected/75756d4a-6197-4dfc-9efb-80cc78fc958d-kube-api-access-4h5zz\") pod \"payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw\" (UID: \"75756d4a-6197-4dfc-9efb-80cc78fc958d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" Oct 30 13:38:00.879331 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.879307 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75756d4a-6197-4dfc-9efb-80cc78fc958d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw\" (UID: \"75756d4a-6197-4dfc-9efb-80cc78fc958d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" Oct 30 13:38:00.903092 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.903059 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h5zz\" (UniqueName: \"kubernetes.io/projected/75756d4a-6197-4dfc-9efb-80cc78fc958d-kube-api-access-4h5zz\") pod \"payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw\" (UID: \"75756d4a-6197-4dfc-9efb-80cc78fc958d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" Oct 30 13:38:00.999319 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:00.999282 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" Oct 30 13:38:01.071026 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:01.070760 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" Oct 30 13:38:01.343847 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:01.343808 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l"] Oct 30 13:38:01.345282 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:38:01.345250 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a04bf8_d29c_425c_b351_61f785e861a3.slice/crio-a9d37a5e8acddac87707dd66205beb7bd89e345e89022926c1ff0ed657237c29 WatchSource:0}: Error finding container a9d37a5e8acddac87707dd66205beb7bd89e345e89022926c1ff0ed657237c29: Status 404 returned error can't find the container with id a9d37a5e8acddac87707dd66205beb7bd89e345e89022926c1ff0ed657237c29 Oct 30 13:38:01.423262 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:01.423226 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw"] Oct 30 13:38:01.423625 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:38:01.423595 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75756d4a_6197_4dfc_9efb_80cc78fc958d.slice/crio-90a152f4bf92e28e027f906bf1dbe9e42ffd4eb5aa5b3ec08c93db4c44b04c15 WatchSource:0}: Error finding container 90a152f4bf92e28e027f906bf1dbe9e42ffd4eb5aa5b3ec08c93db4c44b04c15: Status 404 returned error can't find the container with id 90a152f4bf92e28e027f906bf1dbe9e42ffd4eb5aa5b3ec08c93db4c44b04c15 Oct 30 13:38:01.691245 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:01.691163 7010 generic.go:356] "Generic (PLEG): container finished" podID="a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" containerID="0d30e05091e905667d1a1994a64024b26cd2e7a944a7bb8ee82e87c8be805bc1" exitCode=0 Oct 30 13:38:01.691245 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:01.691233 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" event={"ID":"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53","Type":"ContainerDied","Data":"0d30e05091e905667d1a1994a64024b26cd2e7a944a7bb8ee82e87c8be805bc1"} Oct 30 13:38:01.691806 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:01.691767 7010 scope.go:122] "RemoveContainer" containerID="0d30e05091e905667d1a1994a64024b26cd2e7a944a7bb8ee82e87c8be805bc1" Oct 30 13:38:01.693385 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:01.693355 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" event={"ID":"75756d4a-6197-4dfc-9efb-80cc78fc958d","Type":"ContainerStarted","Data":"97963a5067a4a9242a1dc64d5abb9327cdcc432569eb0ed68778e71c7399a0d3"} Oct 30 13:38:01.693540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:01.693418 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" Oct 30 13:38:01.693540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:01.693450 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" event={"ID":"75756d4a-6197-4dfc-9efb-80cc78fc958d","Type":"ContainerStarted","Data":"90a152f4bf92e28e027f906bf1dbe9e42ffd4eb5aa5b3ec08c93db4c44b04c15"} Oct 30 13:38:01.693996 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:01.693966 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:38:01.694084 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:01.694054 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:38:01.694151 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:01.694103 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:38:01.695768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:01.695478 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" event={"ID":"75a04bf8-d29c-425c-b351-61f785e861a3","Type":"ContainerStarted","Data":"0d001311689fd034400db3c34f9046ae865f4020c153167c2629008f7284e8a6"} Oct 30 13:38:01.695768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:01.695564 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" event={"ID":"75a04bf8-d29c-425c-b351-61f785e861a3","Type":"ContainerStarted","Data":"a9d37a5e8acddac87707dd66205beb7bd89e345e89022926c1ff0ed657237c29"} Oct 30 13:38:01.695768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:01.695595 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" Oct 30 13:38:01.852341 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:01.852272 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" podStartSLOduration=1.852255585 podStartE2EDuration="1.852255585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:01.807581334 +0000 UTC m=+2738.439768529" watchObservedRunningTime="2026-10-30 13:38:01.852255585 +0000 UTC m=+2738.484442746" Oct 30 13:38:01.853957 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:01.853900 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" podStartSLOduration=1.853886269 podStartE2EDuration="1.853886269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:01.849846011 +0000 UTC m=+2738.482033196" watchObservedRunningTime="2026-10-30 13:38:01.853886269 +0000 UTC m=+2738.486073432" Oct 30 13:38:02.704834 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:02.704794 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" event={"ID":"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53","Type":"ContainerStarted","Data":"76e45eb639a8210d98173abdfa875889b824bed2e83eaedc35789ccb760c561a"} Oct 30 13:38:03.170389 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.170329 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" podUID="10326e74-37cd-41d8-b783-a8d1aae34d8e" containerName="payload-processing" containerID="cri-o://5fb25c6ebd4ff8739243f3666237675260745a5727fd77341d8072a7bf865a2f" gracePeriod=25 Oct 30 13:38:03.333476 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.333439 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:38:03.334666 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.334640 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 13:38:03.334796 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.334688 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" podUID="a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 13:38:03.524764 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.524738 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" Oct 30 13:38:03.636345 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.636304 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10326e74-37cd-41d8-b783-a8d1aae34d8e-kube-api-access-xd97q\" (UniqueName: \"kubernetes.io/projected/10326e74-37cd-41d8-b783-a8d1aae34d8e-kube-api-access-xd97q\") pod \"10326e74-37cd-41d8-b783-a8d1aae34d8e\" (UID: \"10326e74-37cd-41d8-b783-a8d1aae34d8e\") " Oct 30 13:38:03.636568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.636376 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10326e74-37cd-41d8-b783-a8d1aae34d8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10326e74-37cd-41d8-b783-a8d1aae34d8e-plugins-config-volume\") pod \"10326e74-37cd-41d8-b783-a8d1aae34d8e\" (UID: \"10326e74-37cd-41d8-b783-a8d1aae34d8e\") " Oct 30 13:38:03.636935 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.636859 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10326e74-37cd-41d8-b783-a8d1aae34d8e-plugins-config-volume" pod "10326e74-37cd-41d8-b783-a8d1aae34d8e" (UID: "10326e74-37cd-41d8-b783-a8d1aae34d8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:03.637336 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.637310 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10326e74-37cd-41d8-b783-a8d1aae34d8e-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:03.639008 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.638979 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10326e74-37cd-41d8-b783-a8d1aae34d8e-kube-api-access-xd97q" pod "10326e74-37cd-41d8-b783-a8d1aae34d8e" (UID: "10326e74-37cd-41d8-b783-a8d1aae34d8e"). InnerVolumeSpecName "kube-api-access-xd97q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:03.717616 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.717474 7010 generic.go:356] "Generic (PLEG): container finished" podID="10326e74-37cd-41d8-b783-a8d1aae34d8e" containerID="5fb25c6ebd4ff8739243f3666237675260745a5727fd77341d8072a7bf865a2f" exitCode=0 Oct 30 13:38:03.717616 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.717539 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" Oct 30 13:38:03.717616 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.717575 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" event={"ID":"10326e74-37cd-41d8-b783-a8d1aae34d8e","Type":"ContainerDied","Data":"5fb25c6ebd4ff8739243f3666237675260745a5727fd77341d8072a7bf865a2f"} Oct 30 13:38:03.718198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.717636 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6" event={"ID":"10326e74-37cd-41d8-b783-a8d1aae34d8e","Type":"ContainerDied","Data":"562e3bdaa26eacd8954423457c5ec788d4ae540331311ff85b43977581c96db3"} Oct 30 13:38:03.718198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.717666 7010 scope.go:122] "RemoveContainer" containerID="5fb25c6ebd4ff8739243f3666237675260745a5727fd77341d8072a7bf865a2f" Oct 30 13:38:03.729061 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.729037 7010 scope.go:122] "RemoveContainer" containerID="5fb25c6ebd4ff8739243f3666237675260745a5727fd77341d8072a7bf865a2f" Oct 30 13:38:03.729361 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:38:03.729340 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb25c6ebd4ff8739243f3666237675260745a5727fd77341d8072a7bf865a2f\": container with ID starting with 5fb25c6ebd4ff8739243f3666237675260745a5727fd77341d8072a7bf865a2f not found: ID does not exist" containerID="5fb25c6ebd4ff8739243f3666237675260745a5727fd77341d8072a7bf865a2f" Oct 30 13:38:03.729433 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.729375 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb25c6ebd4ff8739243f3666237675260745a5727fd77341d8072a7bf865a2f"} err="failed to get container status \"5fb25c6ebd4ff8739243f3666237675260745a5727fd77341d8072a7bf865a2f\": rpc error: code = NotFound desc = could not find container \"5fb25c6ebd4ff8739243f3666237675260745a5727fd77341d8072a7bf865a2f\": container with ID starting with 5fb25c6ebd4ff8739243f3666237675260745a5727fd77341d8072a7bf865a2f not found: ID does not exist" Oct 30 13:38:03.738298 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.738269 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd97q\" (UniqueName: \"kubernetes.io/projected/10326e74-37cd-41d8-b783-a8d1aae34d8e-kube-api-access-xd97q\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:03.776891 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.776855 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6"] Oct 30 13:38:03.779855 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:03.779829 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-675cd445cb-xxhf6"] Oct 30 13:38:04.073001 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:04.072921 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:38:04.333580 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:04.333483 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 13:38:04.333580 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:04.333563 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" podUID="a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 13:38:05.225535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.224535 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj"] Oct 30 13:38:05.237514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.233576 7010 cpu_manager.go:414] "containerMap: removing container" podUID="10326e74-37cd-41d8-b783-a8d1aae34d8e" containerName="payload-processing" Oct 30 13:38:05.237514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.233617 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10326e74-37cd-41d8-b783-a8d1aae34d8e" containerName="payload-processing" Oct 30 13:38:05.237514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.233982 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="10326e74-37cd-41d8-b783-a8d1aae34d8e" containerName="payload-processing" Oct 30 13:38:05.256514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.250682 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj"] Oct 30 13:38:05.256514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.250841 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" Oct 30 13:38:05.298310 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.295911 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-cf3c79-dockercfg-qls66\"" Oct 30 13:38:05.334593 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.334554 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-cf3c79\"" Oct 30 13:38:05.334903 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.334877 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 13:38:05.334995 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.334928 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" podUID="a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 13:38:05.406583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.399076 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad23e4ff-651b-4e54-a180-d1cc7c73c51d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj\" (UID: \"ad23e4ff-651b-4e54-a180-d1cc7c73c51d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" Oct 30 13:38:05.406583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.399158 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm7qh\" (UniqueName: \"kubernetes.io/projected/ad23e4ff-651b-4e54-a180-d1cc7c73c51d-kube-api-access-dm7qh\") pod \"payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj\" (UID: \"ad23e4ff-651b-4e54-a180-d1cc7c73c51d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" Oct 30 13:38:05.406583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.399453 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6"] Oct 30 13:38:05.416521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.415071 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" Oct 30 13:38:05.416521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.416509 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6"] Oct 30 13:38:05.509288 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.509255 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad23e4ff-651b-4e54-a180-d1cc7c73c51d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj\" (UID: \"ad23e4ff-651b-4e54-a180-d1cc7c73c51d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" Oct 30 13:38:05.509535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.509307 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm7qh\" (UniqueName: \"kubernetes.io/projected/ad23e4ff-651b-4e54-a180-d1cc7c73c51d-kube-api-access-dm7qh\") pod \"payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj\" (UID: \"ad23e4ff-651b-4e54-a180-d1cc7c73c51d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" Oct 30 13:38:05.509535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.509412 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51e863c-9c51-4df1-af43-7877a1e2c227-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6\" (UID: \"e51e863c-9c51-4df1-af43-7877a1e2c227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" Oct 30 13:38:05.509535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.509508 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slrlp\" (UniqueName: \"kubernetes.io/projected/e51e863c-9c51-4df1-af43-7877a1e2c227-kube-api-access-slrlp\") pod \"payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6\" (UID: \"e51e863c-9c51-4df1-af43-7877a1e2c227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" Oct 30 13:38:05.510171 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.510138 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad23e4ff-651b-4e54-a180-d1cc7c73c51d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj\" (UID: \"ad23e4ff-651b-4e54-a180-d1cc7c73c51d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" Oct 30 13:38:05.514923 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.514899 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10326e74-37cd-41d8-b783-a8d1aae34d8e" path="/var/lib/kubelet/pods/10326e74-37cd-41d8-b783-a8d1aae34d8e/volumes" Oct 30 13:38:05.534524 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.534482 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm7qh\" (UniqueName: \"kubernetes.io/projected/ad23e4ff-651b-4e54-a180-d1cc7c73c51d-kube-api-access-dm7qh\") pod \"payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj\" (UID: \"ad23e4ff-651b-4e54-a180-d1cc7c73c51d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" Oct 30 13:38:05.608579 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.606308 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" Oct 30 13:38:05.610940 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.610757 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51e863c-9c51-4df1-af43-7877a1e2c227-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6\" (UID: \"e51e863c-9c51-4df1-af43-7877a1e2c227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" Oct 30 13:38:05.610940 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.610851 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slrlp\" (UniqueName: \"kubernetes.io/projected/e51e863c-9c51-4df1-af43-7877a1e2c227-kube-api-access-slrlp\") pod \"payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6\" (UID: \"e51e863c-9c51-4df1-af43-7877a1e2c227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" Oct 30 13:38:05.612335 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.612301 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51e863c-9c51-4df1-af43-7877a1e2c227-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6\" (UID: \"e51e863c-9c51-4df1-af43-7877a1e2c227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" Oct 30 13:38:05.635232 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.635207 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slrlp\" (UniqueName: \"kubernetes.io/projected/e51e863c-9c51-4df1-af43-7877a1e2c227-kube-api-access-slrlp\") pod \"payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6\" (UID: \"e51e863c-9c51-4df1-af43-7877a1e2c227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" Oct 30 13:38:05.738531 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.737477 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" Oct 30 13:38:05.926258 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:05.926214 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj"] Oct 30 13:38:05.927602 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:38:05.927569 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad23e4ff_651b_4e54_a180_d1cc7c73c51d.slice/crio-5c1e66b0f8a65021c9d4b3bd634a890d33a18e092776e06e89d873332c8edd7d WatchSource:0}: Error finding container 5c1e66b0f8a65021c9d4b3bd634a890d33a18e092776e06e89d873332c8edd7d: Status 404 returned error can't find the container with id 5c1e66b0f8a65021c9d4b3bd634a890d33a18e092776e06e89d873332c8edd7d Oct 30 13:38:06.113833 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:06.113799 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6"] Oct 30 13:38:06.116034 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:38:06.116008 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51e863c_9c51_4df1_af43_7877a1e2c227.slice/crio-7f2367792fa5e4bd7ce74396fa927f80069181fc5ca9277731fc80be2885216f WatchSource:0}: Error finding container 7f2367792fa5e4bd7ce74396fa927f80069181fc5ca9277731fc80be2885216f: Status 404 returned error can't find the container with id 7f2367792fa5e4bd7ce74396fa927f80069181fc5ca9277731fc80be2885216f Oct 30 13:38:06.334080 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:06.334046 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 13:38:06.334539 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:06.334093 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" podUID="a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 13:38:06.750590 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:06.750549 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" event={"ID":"ad23e4ff-651b-4e54-a180-d1cc7c73c51d","Type":"ContainerStarted","Data":"b7c05fdb5a7714af87760713f904aceb91f618bc85fee2af41da19cf82941b8d"} Oct 30 13:38:06.750798 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:06.750648 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" event={"ID":"ad23e4ff-651b-4e54-a180-d1cc7c73c51d","Type":"ContainerStarted","Data":"5c1e66b0f8a65021c9d4b3bd634a890d33a18e092776e06e89d873332c8edd7d"} Oct 30 13:38:06.750798 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:06.750779 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" Oct 30 13:38:06.752422 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:06.752396 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" event={"ID":"e51e863c-9c51-4df1-af43-7877a1e2c227","Type":"ContainerStarted","Data":"2a0c3e53b8df63a356fb2c3204e85a9ef29c7acfb714a3539d9dc1b241dd55d0"} Oct 30 13:38:06.752581 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:06.752430 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" event={"ID":"e51e863c-9c51-4df1-af43-7877a1e2c227","Type":"ContainerStarted","Data":"7f2367792fa5e4bd7ce74396fa927f80069181fc5ca9277731fc80be2885216f"} Oct 30 13:38:06.752581 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:06.752552 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" Oct 30 13:38:06.820974 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:06.820906 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" podStartSLOduration=1.820883035 podStartE2EDuration="1.820883035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:06.804886215 +0000 UTC m=+2743.437073377" watchObservedRunningTime="2026-10-30 13:38:06.820883035 +0000 UTC m=+2743.453070198" Oct 30 13:38:06.874811 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:06.874761 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" podStartSLOduration=1.874746312 podStartE2EDuration="1.874746312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:06.871881925 +0000 UTC m=+2743.504069291" watchObservedRunningTime="2026-10-30 13:38:06.874746312 +0000 UTC m=+2743.506933487" Oct 30 13:38:07.334278 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:07.334235 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 13:38:07.334730 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:07.334310 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" podUID="a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 13:38:08.335519 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:08.334054 7010 patch_prober.go:29] interesting pod/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 13:38:08.335519 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:08.334119 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" podUID="a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 13:38:09.339464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:09.339432 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:38:09.340672 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:09.340651 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:38:12.707650 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:12.707617 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" Oct 30 13:38:12.708057 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:12.707700 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" Oct 30 13:38:17.761461 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:17.761418 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" Oct 30 13:38:17.761899 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:17.761647 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" Oct 30 13:38:19.672684 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:19.672655 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l_75a04bf8-d29c-425c-b351-61f785e861a3/payload-processing/0.log" Oct 30 13:38:29.405288 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:38:29.405245 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:38:33.499112 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:33.499072 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc"] Oct 30 13:38:33.499675 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:33.499317 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" podUID="209c4a96-a64e-4251-86cb-80b61f95d020" containerName="payload-processing" containerID="cri-o://104cc2e8425982347b4f51d56b55aa2ce46ff00ea5c16a437b3c06dc64a4d9e9" gracePeriod=30 Oct 30 13:38:33.537202 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:33.537163 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km"] Oct 30 13:38:33.537430 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:33.537402 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" podUID="4252bbb1-b2b1-4e24-9867-7d93421825b5" containerName="payload-pre-processing" containerID="cri-o://a59a43cb5ab05c0d720af46ab658f4029e85fc67351f39202e81fe134cef9862" gracePeriod=30 Oct 30 13:38:34.075614 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.070884 7010 generic.go:356] "Generic (PLEG): container finished" podID="209c4a96-a64e-4251-86cb-80b61f95d020" containerID="104cc2e8425982347b4f51d56b55aa2ce46ff00ea5c16a437b3c06dc64a4d9e9" exitCode=0 Oct 30 13:38:34.075614 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.071132 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" event={"ID":"209c4a96-a64e-4251-86cb-80b61f95d020","Type":"ContainerDied","Data":"104cc2e8425982347b4f51d56b55aa2ce46ff00ea5c16a437b3c06dc64a4d9e9"} Oct 30 13:38:34.096513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.093705 7010 generic.go:356] "Generic (PLEG): container finished" podID="4252bbb1-b2b1-4e24-9867-7d93421825b5" containerID="a59a43cb5ab05c0d720af46ab658f4029e85fc67351f39202e81fe134cef9862" exitCode=0 Oct 30 13:38:34.096513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.093813 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" event={"ID":"4252bbb1-b2b1-4e24-9867-7d93421825b5","Type":"ContainerDied","Data":"a59a43cb5ab05c0d720af46ab658f4029e85fc67351f39202e81fe134cef9862"} Oct 30 13:38:34.106562 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.106531 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" Oct 30 13:38:34.111206 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.108718 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" Oct 30 13:38:34.195854 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.195815 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209c4a96-a64e-4251-86cb-80b61f95d020-kube-api-access-cnkh8\" (UniqueName: \"kubernetes.io/projected/209c4a96-a64e-4251-86cb-80b61f95d020-kube-api-access-cnkh8\") pod \"209c4a96-a64e-4251-86cb-80b61f95d020\" (UID: \"209c4a96-a64e-4251-86cb-80b61f95d020\") " Oct 30 13:38:34.196065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.195970 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4252bbb1-b2b1-4e24-9867-7d93421825b5-kube-api-access-xrg4n\" (UniqueName: \"kubernetes.io/projected/4252bbb1-b2b1-4e24-9867-7d93421825b5-kube-api-access-xrg4n\") pod \"4252bbb1-b2b1-4e24-9867-7d93421825b5\" (UID: \"4252bbb1-b2b1-4e24-9867-7d93421825b5\") " Oct 30 13:38:34.196135 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.196089 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4252bbb1-b2b1-4e24-9867-7d93421825b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4252bbb1-b2b1-4e24-9867-7d93421825b5-plugins-config-volume\") pod \"4252bbb1-b2b1-4e24-9867-7d93421825b5\" (UID: \"4252bbb1-b2b1-4e24-9867-7d93421825b5\") " Oct 30 13:38:34.196190 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.196152 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/209c4a96-a64e-4251-86cb-80b61f95d020-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209c4a96-a64e-4251-86cb-80b61f95d020-plugins-config-volume\") pod \"209c4a96-a64e-4251-86cb-80b61f95d020\" (UID: \"209c4a96-a64e-4251-86cb-80b61f95d020\") " Oct 30 13:38:34.198256 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.197428 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4252bbb1-b2b1-4e24-9867-7d93421825b5-plugins-config-volume" pod "4252bbb1-b2b1-4e24-9867-7d93421825b5" (UID: "4252bbb1-b2b1-4e24-9867-7d93421825b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:34.198256 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.197798 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209c4a96-a64e-4251-86cb-80b61f95d020-plugins-config-volume" pod "209c4a96-a64e-4251-86cb-80b61f95d020" (UID: "209c4a96-a64e-4251-86cb-80b61f95d020"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:34.200740 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.200701 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209c4a96-a64e-4251-86cb-80b61f95d020-kube-api-access-cnkh8" pod "209c4a96-a64e-4251-86cb-80b61f95d020" (UID: "209c4a96-a64e-4251-86cb-80b61f95d020"). InnerVolumeSpecName "kube-api-access-cnkh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:34.201115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.201088 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4252bbb1-b2b1-4e24-9867-7d93421825b5-kube-api-access-xrg4n" pod "4252bbb1-b2b1-4e24-9867-7d93421825b5" (UID: "4252bbb1-b2b1-4e24-9867-7d93421825b5"). InnerVolumeSpecName "kube-api-access-xrg4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:34.298543 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.298469 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4252bbb1-b2b1-4e24-9867-7d93421825b5-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:34.298543 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.298546 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209c4a96-a64e-4251-86cb-80b61f95d020-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:34.298806 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.298562 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnkh8\" (UniqueName: \"kubernetes.io/projected/209c4a96-a64e-4251-86cb-80b61f95d020-kube-api-access-cnkh8\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:34.298806 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:34.298576 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrg4n\" (UniqueName: \"kubernetes.io/projected/4252bbb1-b2b1-4e24-9867-7d93421825b5-kube-api-access-xrg4n\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:35.101842 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:35.101813 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" Oct 30 13:38:35.102262 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:35.101852 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc" event={"ID":"209c4a96-a64e-4251-86cb-80b61f95d020","Type":"ContainerDied","Data":"c6f01f26569fcfb0beda9f81a61d32ddb9691af25b2914fceb9046c453889730"} Oct 30 13:38:35.102262 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:35.101919 7010 scope.go:122] "RemoveContainer" containerID="104cc2e8425982347b4f51d56b55aa2ce46ff00ea5c16a437b3c06dc64a4d9e9" Oct 30 13:38:35.103532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:35.103509 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" Oct 30 13:38:35.103532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:35.103525 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km" event={"ID":"4252bbb1-b2b1-4e24-9867-7d93421825b5","Type":"ContainerDied","Data":"8475611a67f61751a158830d374ec147e50c34d84fdff5423a73dfc4017787e4"} Oct 30 13:38:35.113599 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:35.113559 7010 scope.go:122] "RemoveContainer" containerID="a59a43cb5ab05c0d720af46ab658f4029e85fc67351f39202e81fe134cef9862" Oct 30 13:38:35.155923 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:35.155891 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc"] Oct 30 13:38:35.168519 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:35.168479 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-082748-545497b8b8-jkszc"] Oct 30 13:38:35.198169 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:35.198135 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km"] Oct 30 13:38:35.202534 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:35.202510 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-082748-d6cdc5d9f-8r8km"] Oct 30 13:38:35.489336 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:35.489302 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="209c4a96-a64e-4251-86cb-80b61f95d020" path="/var/lib/kubelet/pods/209c4a96-a64e-4251-86cb-80b61f95d020/volumes" Oct 30 13:38:35.489812 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:35.489796 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4252bbb1-b2b1-4e24-9867-7d93421825b5" path="/var/lib/kubelet/pods/4252bbb1-b2b1-4e24-9867-7d93421825b5/volumes" Oct 30 13:38:42.140860 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:42.139895 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9"] Oct 30 13:38:42.140860 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:42.140401 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" podUID="915ff449-5d83-4605-9ab8-e2ba528028a8" containerName="istio-proxy" containerID="cri-o://30a989274d796ecb0574d2c6ae18fe8cf5b25052937ef787ba0b76a0bc938e7f" gracePeriod=30 Oct 30 13:38:42.155876 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:42.155841 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt"] Oct 30 13:38:42.157260 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:42.157206 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" podUID="1589354d-4858-4ad9-a76f-305adc4c83a9" containerName="istio-proxy" containerID="cri-o://8335bdfeb31c5f48284b621f49385bac23f9882d60406e5b21fa513f22315159" gracePeriod=30 Oct 30 13:38:47.459362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.459328 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:38:47.504722 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.504690 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:38:47.574107 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.574065 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-credential-socket\") pod \"1589354d-4858-4ad9-a76f-305adc4c83a9\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " Oct 30 13:38:47.574309 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.574137 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-workload-socket\") pod \"1589354d-4858-4ad9-a76f-305adc4c83a9\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " Oct 30 13:38:47.574309 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.574183 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-credential-socket\") pod \"915ff449-5d83-4605-9ab8-e2ba528028a8\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " Oct 30 13:38:47.574309 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.574227 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-data\") pod \"915ff449-5d83-4605-9ab8-e2ba528028a8\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " Oct 30 13:38:47.574309 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.574281 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/915ff449-5d83-4605-9ab8-e2ba528028a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915ff449-5d83-4605-9ab8-e2ba528028a8-istiod-ca-cert\") pod \"915ff449-5d83-4605-9ab8-e2ba528028a8\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " Oct 30 13:38:47.574548 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.574314 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1589354d-4858-4ad9-a76f-305adc4c83a9-kube-api-access-kh6xm\" (UniqueName: \"kubernetes.io/projected/1589354d-4858-4ad9-a76f-305adc4c83a9-kube-api-access-kh6xm\") pod \"1589354d-4858-4ad9-a76f-305adc4c83a9\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " Oct 30 13:38:47.574548 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.574344 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-podinfo\") pod \"915ff449-5d83-4605-9ab8-e2ba528028a8\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " Oct 30 13:38:47.574548 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.574401 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-token\" (UniqueName: \"kubernetes.io/projected/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-token\") pod \"1589354d-4858-4ad9-a76f-305adc4c83a9\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " Oct 30 13:38:47.574548 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.574448 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-workload-certs\") pod \"915ff449-5d83-4605-9ab8-e2ba528028a8\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " Oct 30 13:38:47.574548 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.574475 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-token\" (UniqueName: \"kubernetes.io/projected/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-token\") pod \"915ff449-5d83-4605-9ab8-e2ba528028a8\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " Oct 30 13:38:47.574799 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.574616 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-workload-certs\") pod \"1589354d-4858-4ad9-a76f-305adc4c83a9\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " Oct 30 13:38:47.574799 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.574656 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1589354d-4858-4ad9-a76f-305adc4c83a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1589354d-4858-4ad9-a76f-305adc4c83a9-istiod-ca-cert\") pod \"1589354d-4858-4ad9-a76f-305adc4c83a9\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " Oct 30 13:38:47.574799 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.574709 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-data\") pod \"1589354d-4858-4ad9-a76f-305adc4c83a9\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " Oct 30 13:38:47.574951 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.574806 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-envoy\") pod \"1589354d-4858-4ad9-a76f-305adc4c83a9\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " Oct 30 13:38:47.574951 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.574833 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/915ff449-5d83-4605-9ab8-e2ba528028a8-kube-api-access-vhm7s\" (UniqueName: \"kubernetes.io/projected/915ff449-5d83-4605-9ab8-e2ba528028a8-kube-api-access-vhm7s\") pod \"915ff449-5d83-4605-9ab8-e2ba528028a8\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " Oct 30 13:38:47.574951 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.574882 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-podinfo\") pod \"1589354d-4858-4ad9-a76f-305adc4c83a9\" (UID: \"1589354d-4858-4ad9-a76f-305adc4c83a9\") " Oct 30 13:38:47.575094 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.574956 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-workload-socket\") pod \"915ff449-5d83-4605-9ab8-e2ba528028a8\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " Oct 30 13:38:47.575094 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.575048 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-envoy\") pod \"915ff449-5d83-4605-9ab8-e2ba528028a8\" (UID: \"915ff449-5d83-4605-9ab8-e2ba528028a8\") " Oct 30 13:38:47.580042 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.580004 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-workload-certs" pod "915ff449-5d83-4605-9ab8-e2ba528028a8" (UID: "915ff449-5d83-4605-9ab8-e2ba528028a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:47.580526 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.580479 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-credential-socket" pod "1589354d-4858-4ad9-a76f-305adc4c83a9" (UID: "1589354d-4858-4ad9-a76f-305adc4c83a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:47.580941 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.580919 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-workload-socket" pod "1589354d-4858-4ad9-a76f-305adc4c83a9" (UID: "1589354d-4858-4ad9-a76f-305adc4c83a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:47.581306 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.581287 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-credential-socket" pod "915ff449-5d83-4605-9ab8-e2ba528028a8" (UID: "915ff449-5d83-4605-9ab8-e2ba528028a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:47.581645 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.581624 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-data" pod "915ff449-5d83-4605-9ab8-e2ba528028a8" (UID: "915ff449-5d83-4605-9ab8-e2ba528028a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:47.582070 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.582051 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/915ff449-5d83-4605-9ab8-e2ba528028a8-istiod-ca-cert" pod "915ff449-5d83-4605-9ab8-e2ba528028a8" (UID: "915ff449-5d83-4605-9ab8-e2ba528028a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:47.592635 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.592596 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1589354d-4858-4ad9-a76f-305adc4c83a9-kube-api-access-kh6xm" pod "1589354d-4858-4ad9-a76f-305adc4c83a9" (UID: "1589354d-4858-4ad9-a76f-305adc4c83a9"). InnerVolumeSpecName "kube-api-access-kh6xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:47.593225 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.593186 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-workload-socket" pod "915ff449-5d83-4605-9ab8-e2ba528028a8" (UID: "915ff449-5d83-4605-9ab8-e2ba528028a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:47.594614 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.594386 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-workload-certs" pod "1589354d-4858-4ad9-a76f-305adc4c83a9" (UID: "1589354d-4858-4ad9-a76f-305adc4c83a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:47.594837 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.594812 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1589354d-4858-4ad9-a76f-305adc4c83a9-istiod-ca-cert" pod "1589354d-4858-4ad9-a76f-305adc4c83a9" (UID: "1589354d-4858-4ad9-a76f-305adc4c83a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:47.595273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.595240 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-envoy" pod "1589354d-4858-4ad9-a76f-305adc4c83a9" (UID: "1589354d-4858-4ad9-a76f-305adc4c83a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:47.595859 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.595828 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915ff449-5d83-4605-9ab8-e2ba528028a8-kube-api-access-vhm7s" pod "915ff449-5d83-4605-9ab8-e2ba528028a8" (UID: "915ff449-5d83-4605-9ab8-e2ba528028a8"). InnerVolumeSpecName "kube-api-access-vhm7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:47.596438 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.596413 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-envoy" pod "915ff449-5d83-4605-9ab8-e2ba528028a8" (UID: "915ff449-5d83-4605-9ab8-e2ba528028a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:47.596626 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.596598 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-data" pod "1589354d-4858-4ad9-a76f-305adc4c83a9" (UID: "1589354d-4858-4ad9-a76f-305adc4c83a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:38:47.596710 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.596682 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-token" pod "915ff449-5d83-4605-9ab8-e2ba528028a8" (UID: "915ff449-5d83-4605-9ab8-e2ba528028a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:47.596782 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.596712 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-token" pod "1589354d-4858-4ad9-a76f-305adc4c83a9" (UID: "1589354d-4858-4ad9-a76f-305adc4c83a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:47.597044 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.597018 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-podinfo" pod "1589354d-4858-4ad9-a76f-305adc4c83a9" (UID: "1589354d-4858-4ad9-a76f-305adc4c83a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:38:47.597917 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.597894 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-podinfo" pod "915ff449-5d83-4605-9ab8-e2ba528028a8" (UID: "915ff449-5d83-4605-9ab8-e2ba528028a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:38:47.677250 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677152 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.677250 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677188 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1589354d-4858-4ad9-a76f-305adc4c83a9-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.677250 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677201 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.677250 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677212 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.677250 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677226 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhm7s\" (UniqueName: \"kubernetes.io/projected/915ff449-5d83-4605-9ab8-e2ba528028a8-kube-api-access-vhm7s\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.677250 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677237 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.677250 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677249 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.677664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677263 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.677664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677275 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.677664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677288 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1589354d-4858-4ad9-a76f-305adc4c83a9-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.677664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677300 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.677664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677310 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.677664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677322 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/915ff449-5d83-4605-9ab8-e2ba528028a8-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.677664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677333 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh6xm\" (UniqueName: \"kubernetes.io/projected/1589354d-4858-4ad9-a76f-305adc4c83a9-kube-api-access-kh6xm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.677664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677344 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.677664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677354 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1589354d-4858-4ad9-a76f-305adc4c83a9-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.677664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677366 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/915ff449-5d83-4605-9ab8-e2ba528028a8-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:47.677664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:47.677377 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/915ff449-5d83-4605-9ab8-e2ba528028a8-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:48.216036 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.216001 7010 generic.go:356] "Generic (PLEG): container finished" podID="1589354d-4858-4ad9-a76f-305adc4c83a9" containerID="8335bdfeb31c5f48284b621f49385bac23f9882d60406e5b21fa513f22315159" exitCode=0 Oct 30 13:38:48.216236 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.216053 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" Oct 30 13:38:48.216236 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.216089 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" event={"ID":"1589354d-4858-4ad9-a76f-305adc4c83a9","Type":"ContainerDied","Data":"8335bdfeb31c5f48284b621f49385bac23f9882d60406e5b21fa513f22315159"} Oct 30 13:38:48.216236 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.216152 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt" event={"ID":"1589354d-4858-4ad9-a76f-305adc4c83a9","Type":"ContainerDied","Data":"3ab48d343f72796b4d61c5a90f69bac367f05d9ebf9b724f55b30f31952a4284"} Oct 30 13:38:48.216236 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.216183 7010 scope.go:122] "RemoveContainer" containerID="8335bdfeb31c5f48284b621f49385bac23f9882d60406e5b21fa513f22315159" Oct 30 13:38:48.217657 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.217632 7010 generic.go:356] "Generic (PLEG): container finished" podID="915ff449-5d83-4605-9ab8-e2ba528028a8" containerID="30a989274d796ecb0574d2c6ae18fe8cf5b25052937ef787ba0b76a0bc938e7f" exitCode=0 Oct 30 13:38:48.217748 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.217678 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" Oct 30 13:38:48.217748 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.217713 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" event={"ID":"915ff449-5d83-4605-9ab8-e2ba528028a8","Type":"ContainerDied","Data":"30a989274d796ecb0574d2c6ae18fe8cf5b25052937ef787ba0b76a0bc938e7f"} Oct 30 13:38:48.217851 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.217764 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9" event={"ID":"915ff449-5d83-4605-9ab8-e2ba528028a8","Type":"ContainerDied","Data":"b1ff9723e3bf4ae6b21d73fc26ea3872d40d225cdeae6307c27fe68030e6d3eb"} Oct 30 13:38:48.227837 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.227812 7010 scope.go:122] "RemoveContainer" containerID="96eb82a94d4f48ae413198095b2aeede03c013b977864e0efff8d18c2d68dcf1" Oct 30 13:38:48.239809 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.239785 7010 scope.go:122] "RemoveContainer" containerID="8335bdfeb31c5f48284b621f49385bac23f9882d60406e5b21fa513f22315159" Oct 30 13:38:48.240155 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:38:48.240133 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8335bdfeb31c5f48284b621f49385bac23f9882d60406e5b21fa513f22315159\": container with ID starting with 8335bdfeb31c5f48284b621f49385bac23f9882d60406e5b21fa513f22315159 not found: ID does not exist" containerID="8335bdfeb31c5f48284b621f49385bac23f9882d60406e5b21fa513f22315159" Oct 30 13:38:48.240205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.240169 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8335bdfeb31c5f48284b621f49385bac23f9882d60406e5b21fa513f22315159"} err="failed to get container status \"8335bdfeb31c5f48284b621f49385bac23f9882d60406e5b21fa513f22315159\": rpc error: code = NotFound desc = could not find container \"8335bdfeb31c5f48284b621f49385bac23f9882d60406e5b21fa513f22315159\": container with ID starting with 8335bdfeb31c5f48284b621f49385bac23f9882d60406e5b21fa513f22315159 not found: ID does not exist" Oct 30 13:38:48.240205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.240187 7010 scope.go:122] "RemoveContainer" containerID="96eb82a94d4f48ae413198095b2aeede03c013b977864e0efff8d18c2d68dcf1" Oct 30 13:38:48.240443 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:38:48.240424 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96eb82a94d4f48ae413198095b2aeede03c013b977864e0efff8d18c2d68dcf1\": container with ID starting with 96eb82a94d4f48ae413198095b2aeede03c013b977864e0efff8d18c2d68dcf1 not found: ID does not exist" containerID="96eb82a94d4f48ae413198095b2aeede03c013b977864e0efff8d18c2d68dcf1" Oct 30 13:38:48.240512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.240449 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96eb82a94d4f48ae413198095b2aeede03c013b977864e0efff8d18c2d68dcf1"} err="failed to get container status \"96eb82a94d4f48ae413198095b2aeede03c013b977864e0efff8d18c2d68dcf1\": rpc error: code = NotFound desc = could not find container \"96eb82a94d4f48ae413198095b2aeede03c013b977864e0efff8d18c2d68dcf1\": container with ID starting with 96eb82a94d4f48ae413198095b2aeede03c013b977864e0efff8d18c2d68dcf1 not found: ID does not exist" Oct 30 13:38:48.240512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.240468 7010 scope.go:122] "RemoveContainer" containerID="30a989274d796ecb0574d2c6ae18fe8cf5b25052937ef787ba0b76a0bc938e7f" Oct 30 13:38:48.251538 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.251506 7010 scope.go:122] "RemoveContainer" containerID="30a989274d796ecb0574d2c6ae18fe8cf5b25052937ef787ba0b76a0bc938e7f" Oct 30 13:38:48.251927 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:38:48.251900 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a989274d796ecb0574d2c6ae18fe8cf5b25052937ef787ba0b76a0bc938e7f\": container with ID starting with 30a989274d796ecb0574d2c6ae18fe8cf5b25052937ef787ba0b76a0bc938e7f not found: ID does not exist" containerID="30a989274d796ecb0574d2c6ae18fe8cf5b25052937ef787ba0b76a0bc938e7f" Oct 30 13:38:48.252032 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.251939 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a989274d796ecb0574d2c6ae18fe8cf5b25052937ef787ba0b76a0bc938e7f"} err="failed to get container status \"30a989274d796ecb0574d2c6ae18fe8cf5b25052937ef787ba0b76a0bc938e7f\": rpc error: code = NotFound desc = could not find container \"30a989274d796ecb0574d2c6ae18fe8cf5b25052937ef787ba0b76a0bc938e7f\": container with ID starting with 30a989274d796ecb0574d2c6ae18fe8cf5b25052937ef787ba0b76a0bc938e7f not found: ID does not exist" Oct 30 13:38:48.278546 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.278474 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt"] Oct 30 13:38:48.290198 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.290160 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-dxsbt"] Oct 30 13:38:48.319739 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.319702 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9"] Oct 30 13:38:48.323284 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:48.323253 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-082748-openshift-default-d564998c7-bkbn9"] Oct 30 13:38:49.490819 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:49.490783 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1589354d-4858-4ad9-a76f-305adc4c83a9" path="/var/lib/kubelet/pods/1589354d-4858-4ad9-a76f-305adc4c83a9/volumes" Oct 30 13:38:49.491359 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:49.491341 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="915ff449-5d83-4605-9ab8-e2ba528028a8" path="/var/lib/kubelet/pods/915ff449-5d83-4605-9ab8-e2ba528028a8/volumes" Oct 30 13:38:52.276394 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.276137 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97"] Oct 30 13:38:52.278308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.278273 7010 cpu_manager.go:414] "containerMap: removing container" podUID="915ff449-5d83-4605-9ab8-e2ba528028a8" containerName="istio-proxy" Oct 30 13:38:52.278308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.278307 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="915ff449-5d83-4605-9ab8-e2ba528028a8" containerName="istio-proxy" Oct 30 13:38:52.278588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.278354 7010 cpu_manager.go:414] "containerMap: removing container" podUID="1589354d-4858-4ad9-a76f-305adc4c83a9" containerName="istio-proxy" Oct 30 13:38:52.278588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.278363 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1589354d-4858-4ad9-a76f-305adc4c83a9" containerName="istio-proxy" Oct 30 13:38:52.278588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.278381 7010 cpu_manager.go:414] "containerMap: removing container" podUID="209c4a96-a64e-4251-86cb-80b61f95d020" containerName="payload-processing" Oct 30 13:38:52.278588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.278389 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209c4a96-a64e-4251-86cb-80b61f95d020" containerName="payload-processing" Oct 30 13:38:52.278588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.278417 7010 cpu_manager.go:414] "containerMap: removing container" podUID="4252bbb1-b2b1-4e24-9867-7d93421825b5" containerName="payload-pre-processing" Oct 30 13:38:52.278588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.278425 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4252bbb1-b2b1-4e24-9867-7d93421825b5" containerName="payload-pre-processing" Oct 30 13:38:52.278924 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.278622 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="915ff449-5d83-4605-9ab8-e2ba528028a8" containerName="istio-proxy" Oct 30 13:38:52.278924 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.278650 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="1589354d-4858-4ad9-a76f-305adc4c83a9" containerName="istio-proxy" Oct 30 13:38:52.278924 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.278662 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="209c4a96-a64e-4251-86cb-80b61f95d020" containerName="payload-processing" Oct 30 13:38:52.278924 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.278674 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="1589354d-4858-4ad9-a76f-305adc4c83a9" containerName="istio-proxy" Oct 30 13:38:52.278924 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.278686 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="4252bbb1-b2b1-4e24-9867-7d93421825b5" containerName="payload-pre-processing" Oct 30 13:38:52.285003 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.284962 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" Oct 30 13:38:52.303058 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.302999 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97"] Oct 30 13:38:52.311007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.310945 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a8d3b8-dockercfg-qk286\"" Oct 30 13:38:52.311708 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.311349 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a8d3b8\"" Oct 30 13:38:52.423297 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.400020 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxzw2\" (UniqueName: \"kubernetes.io/projected/5972caf5-43d7-4006-8c85-00d82f960f84-kube-api-access-kxzw2\") pod \"payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97\" (UID: \"5972caf5-43d7-4006-8c85-00d82f960f84\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" Oct 30 13:38:52.423297 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.400082 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5972caf5-43d7-4006-8c85-00d82f960f84-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97\" (UID: \"5972caf5-43d7-4006-8c85-00d82f960f84\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" Oct 30 13:38:52.423297 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.418383 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55"] Oct 30 13:38:52.423297 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.420238 7010 cpu_manager.go:414] "containerMap: removing container" podUID="1589354d-4858-4ad9-a76f-305adc4c83a9" containerName="istio-proxy" Oct 30 13:38:52.423297 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.420261 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1589354d-4858-4ad9-a76f-305adc4c83a9" containerName="istio-proxy" Oct 30 13:38:52.425042 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.425006 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" Oct 30 13:38:52.451606 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.451570 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55"] Oct 30 13:38:52.501163 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.501128 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1352f2-5503-4593-96d1-2cbbf798145e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55\" (UID: \"0f1352f2-5503-4593-96d1-2cbbf798145e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" Oct 30 13:38:52.501364 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.501287 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl5gj\" (UniqueName: \"kubernetes.io/projected/0f1352f2-5503-4593-96d1-2cbbf798145e-kube-api-access-fl5gj\") pod \"payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55\" (UID: \"0f1352f2-5503-4593-96d1-2cbbf798145e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" Oct 30 13:38:52.501468 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.501428 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxzw2\" (UniqueName: \"kubernetes.io/projected/5972caf5-43d7-4006-8c85-00d82f960f84-kube-api-access-kxzw2\") pod \"payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97\" (UID: \"5972caf5-43d7-4006-8c85-00d82f960f84\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" Oct 30 13:38:52.501611 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.501524 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5972caf5-43d7-4006-8c85-00d82f960f84-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97\" (UID: \"5972caf5-43d7-4006-8c85-00d82f960f84\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" Oct 30 13:38:52.502381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.502358 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5972caf5-43d7-4006-8c85-00d82f960f84-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97\" (UID: \"5972caf5-43d7-4006-8c85-00d82f960f84\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" Oct 30 13:38:52.525787 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.525753 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxzw2\" (UniqueName: \"kubernetes.io/projected/5972caf5-43d7-4006-8c85-00d82f960f84-kube-api-access-kxzw2\") pod \"payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97\" (UID: \"5972caf5-43d7-4006-8c85-00d82f960f84\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" Oct 30 13:38:52.603258 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.603146 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl5gj\" (UniqueName: \"kubernetes.io/projected/0f1352f2-5503-4593-96d1-2cbbf798145e-kube-api-access-fl5gj\") pod \"payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55\" (UID: \"0f1352f2-5503-4593-96d1-2cbbf798145e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" Oct 30 13:38:52.603560 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.603536 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1352f2-5503-4593-96d1-2cbbf798145e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55\" (UID: \"0f1352f2-5503-4593-96d1-2cbbf798145e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" Oct 30 13:38:52.604741 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.604679 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1352f2-5503-4593-96d1-2cbbf798145e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55\" (UID: \"0f1352f2-5503-4593-96d1-2cbbf798145e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" Oct 30 13:38:52.624573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.624532 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" Oct 30 13:38:52.628235 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.627897 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl5gj\" (UniqueName: \"kubernetes.io/projected/0f1352f2-5503-4593-96d1-2cbbf798145e-kube-api-access-fl5gj\") pod \"payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55\" (UID: \"0f1352f2-5503-4593-96d1-2cbbf798145e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" Oct 30 13:38:52.749473 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.749426 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" Oct 30 13:38:52.936422 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:52.936390 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97"] Oct 30 13:38:52.937335 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:38:52.937307 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5972caf5_43d7_4006_8c85_00d82f960f84.slice/crio-460f7a2987fc4d843f27f643217ef6ee8392bc62d05eb56b9831a0d9044bb107 WatchSource:0}: Error finding container 460f7a2987fc4d843f27f643217ef6ee8392bc62d05eb56b9831a0d9044bb107: Status 404 returned error can't find the container with id 460f7a2987fc4d843f27f643217ef6ee8392bc62d05eb56b9831a0d9044bb107 Oct 30 13:38:53.070537 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:53.070508 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55"] Oct 30 13:38:53.071986 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:38:53.071958 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1352f2_5503_4593_96d1_2cbbf798145e.slice/crio-6514b81992372020c5cdcdbeb81e32cb4d6a0c7c4cb13f8a6ceaa7a7f60d86b8 WatchSource:0}: Error finding container 6514b81992372020c5cdcdbeb81e32cb4d6a0c7c4cb13f8a6ceaa7a7f60d86b8: Status 404 returned error can't find the container with id 6514b81992372020c5cdcdbeb81e32cb4d6a0c7c4cb13f8a6ceaa7a7f60d86b8 Oct 30 13:38:53.260373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:53.260265 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" event={"ID":"0f1352f2-5503-4593-96d1-2cbbf798145e","Type":"ContainerStarted","Data":"848e9085c854bb1ff69884ea2a1137420212f0224a1d1db854238b9ef56d097b"} Oct 30 13:38:53.260373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:53.260333 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" Oct 30 13:38:53.260373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:53.260360 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" event={"ID":"0f1352f2-5503-4593-96d1-2cbbf798145e","Type":"ContainerStarted","Data":"6514b81992372020c5cdcdbeb81e32cb4d6a0c7c4cb13f8a6ceaa7a7f60d86b8"} Oct 30 13:38:53.262061 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:53.262033 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" event={"ID":"5972caf5-43d7-4006-8c85-00d82f960f84","Type":"ContainerStarted","Data":"f3a26631ff753d3a7a725ebab12d570a8ed2380821fcca5b4813d4250b37930d"} Oct 30 13:38:53.262193 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:53.262085 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" event={"ID":"5972caf5-43d7-4006-8c85-00d82f960f84","Type":"ContainerStarted","Data":"460f7a2987fc4d843f27f643217ef6ee8392bc62d05eb56b9831a0d9044bb107"} Oct 30 13:38:53.262193 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:53.262113 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" Oct 30 13:38:53.305453 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:53.305379 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" podStartSLOduration=1.305356373 podStartE2EDuration="1.305356373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:53.302754863 +0000 UTC m=+2789.934942027" watchObservedRunningTime="2026-10-30 13:38:53.305356373 +0000 UTC m=+2789.937543537" Oct 30 13:38:53.348109 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:53.348031 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" podStartSLOduration=1.348009422 podStartE2EDuration="1.348009422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:53.344861352 +0000 UTC m=+2789.977048515" watchObservedRunningTime="2026-10-30 13:38:53.348009422 +0000 UTC m=+2789.980196591" Oct 30 13:38:57.454160 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:57.454123 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj"] Oct 30 13:38:57.454605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:57.454373 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" podUID="5f34026b-12db-4043-a58f-6d0344549ae7" containerName="maas-api" containerID="cri-o://7e1138c0afa7522f730c3f25e593181892c1c97b981d9ce65dee62f41a4bb060" gracePeriod=30 Oct 30 13:38:57.850108 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:57.850064 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l"] Oct 30 13:38:57.850334 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:57.850307 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" podUID="75a04bf8-d29c-425c-b351-61f785e861a3" containerName="payload-processing" containerID="cri-o://0d001311689fd034400db3c34f9046ae865f4020c153167c2629008f7284e8a6" gracePeriod=30 Oct 30 13:38:57.863218 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:57.863186 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:38:57.883590 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:57.875398 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw"] Oct 30 13:38:57.883590 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:57.875706 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" podUID="75756d4a-6197-4dfc-9efb-80cc78fc958d" containerName="payload-pre-processing" containerID="cri-o://97963a5067a4a9242a1dc64d5abb9327cdcc432569eb0ed68778e71c7399a0d3" gracePeriod=30 Oct 30 13:38:58.038509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.038272 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f34026b-12db-4043-a58f-6d0344549ae7-kube-api-access-ktnsz\" (UniqueName: \"kubernetes.io/projected/5f34026b-12db-4043-a58f-6d0344549ae7-kube-api-access-ktnsz\") pod \"5f34026b-12db-4043-a58f-6d0344549ae7\" (UID: \"5f34026b-12db-4043-a58f-6d0344549ae7\") " Oct 30 13:38:58.038509 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.038482 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f34026b-12db-4043-a58f-6d0344549ae7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f34026b-12db-4043-a58f-6d0344549ae7-maas-api-tls\") pod \"5f34026b-12db-4043-a58f-6d0344549ae7\" (UID: \"5f34026b-12db-4043-a58f-6d0344549ae7\") " Oct 30 13:38:58.038753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.038557 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f34026b-12db-4043-a58f-6d0344549ae7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f34026b-12db-4043-a58f-6d0344549ae7-openshift-service-ca\") pod \"5f34026b-12db-4043-a58f-6d0344549ae7\" (UID: \"5f34026b-12db-4043-a58f-6d0344549ae7\") " Oct 30 13:38:58.038875 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.038794 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f34026b-12db-4043-a58f-6d0344549ae7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f34026b-12db-4043-a58f-6d0344549ae7-metrics-certs\") pod \"5f34026b-12db-4043-a58f-6d0344549ae7\" (UID: \"5f34026b-12db-4043-a58f-6d0344549ae7\") " Oct 30 13:38:58.042661 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.042621 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f34026b-12db-4043-a58f-6d0344549ae7-metrics-certs" pod "5f34026b-12db-4043-a58f-6d0344549ae7" (UID: "5f34026b-12db-4043-a58f-6d0344549ae7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:38:58.043031 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.043005 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f34026b-12db-4043-a58f-6d0344549ae7-openshift-service-ca" pod "5f34026b-12db-4043-a58f-6d0344549ae7" (UID: "5f34026b-12db-4043-a58f-6d0344549ae7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:58.044796 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.044764 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f34026b-12db-4043-a58f-6d0344549ae7-maas-api-tls" pod "5f34026b-12db-4043-a58f-6d0344549ae7" (UID: "5f34026b-12db-4043-a58f-6d0344549ae7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:38:58.046758 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.046722 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f34026b-12db-4043-a58f-6d0344549ae7-kube-api-access-ktnsz" pod "5f34026b-12db-4043-a58f-6d0344549ae7" (UID: "5f34026b-12db-4043-a58f-6d0344549ae7"). InnerVolumeSpecName "kube-api-access-ktnsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:58.141456 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.141416 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktnsz\" (UniqueName: \"kubernetes.io/projected/5f34026b-12db-4043-a58f-6d0344549ae7-kube-api-access-ktnsz\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:58.141672 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.141463 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f34026b-12db-4043-a58f-6d0344549ae7-maas-api-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:58.141672 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.141482 7010 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f34026b-12db-4043-a58f-6d0344549ae7-openshift-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:58.141672 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.141575 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f34026b-12db-4043-a58f-6d0344549ae7-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:58.356951 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.356532 7010 generic.go:356] "Generic (PLEG): container finished" podID="75756d4a-6197-4dfc-9efb-80cc78fc958d" containerID="97963a5067a4a9242a1dc64d5abb9327cdcc432569eb0ed68778e71c7399a0d3" exitCode=0 Oct 30 13:38:58.356951 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.356639 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" event={"ID":"75756d4a-6197-4dfc-9efb-80cc78fc958d","Type":"ContainerDied","Data":"97963a5067a4a9242a1dc64d5abb9327cdcc432569eb0ed68778e71c7399a0d3"} Oct 30 13:38:58.356951 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.356688 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" event={"ID":"75756d4a-6197-4dfc-9efb-80cc78fc958d","Type":"ContainerDied","Data":"90a152f4bf92e28e027f906bf1dbe9e42ffd4eb5aa5b3ec08c93db4c44b04c15"} Oct 30 13:38:58.356951 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.356709 7010 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90a152f4bf92e28e027f906bf1dbe9e42ffd4eb5aa5b3ec08c93db4c44b04c15" Oct 30 13:38:58.392052 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.392012 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" Oct 30 13:38:58.395944 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.395909 7010 generic.go:356] "Generic (PLEG): container finished" podID="75a04bf8-d29c-425c-b351-61f785e861a3" containerID="0d001311689fd034400db3c34f9046ae865f4020c153167c2629008f7284e8a6" exitCode=0 Oct 30 13:38:58.396199 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.396177 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" event={"ID":"75a04bf8-d29c-425c-b351-61f785e861a3","Type":"ContainerDied","Data":"0d001311689fd034400db3c34f9046ae865f4020c153167c2629008f7284e8a6"} Oct 30 13:38:58.400147 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.400102 7010 generic.go:356] "Generic (PLEG): container finished" podID="5f34026b-12db-4043-a58f-6d0344549ae7" containerID="7e1138c0afa7522f730c3f25e593181892c1c97b981d9ce65dee62f41a4bb060" exitCode=0 Oct 30 13:38:58.400301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.400165 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" event={"ID":"5f34026b-12db-4043-a58f-6d0344549ae7","Type":"ContainerDied","Data":"7e1138c0afa7522f730c3f25e593181892c1c97b981d9ce65dee62f41a4bb060"} Oct 30 13:38:58.400301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.400215 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" event={"ID":"5f34026b-12db-4043-a58f-6d0344549ae7","Type":"ContainerDied","Data":"d866a01ff74b7fe79c5c3bc4d55ef41317823641807857888c36cffd56b38502"} Oct 30 13:38:58.400301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.400244 7010 scope.go:122] "RemoveContainer" containerID="7e1138c0afa7522f730c3f25e593181892c1c97b981d9ce65dee62f41a4bb060" Oct 30 13:38:58.400301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.400280 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj" Oct 30 13:38:58.409573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.408609 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" Oct 30 13:38:58.413923 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.413894 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75756d4a-6197-4dfc-9efb-80cc78fc958d-kube-api-access-4h5zz\" (UniqueName: \"kubernetes.io/projected/75756d4a-6197-4dfc-9efb-80cc78fc958d-kube-api-access-4h5zz\") pod \"75756d4a-6197-4dfc-9efb-80cc78fc958d\" (UID: \"75756d4a-6197-4dfc-9efb-80cc78fc958d\") " Oct 30 13:38:58.414061 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.414048 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75756d4a-6197-4dfc-9efb-80cc78fc958d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75756d4a-6197-4dfc-9efb-80cc78fc958d-plugins-config-volume\") pod \"75756d4a-6197-4dfc-9efb-80cc78fc958d\" (UID: \"75756d4a-6197-4dfc-9efb-80cc78fc958d\") " Oct 30 13:38:58.414473 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.414442 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75756d4a-6197-4dfc-9efb-80cc78fc958d-plugins-config-volume" pod "75756d4a-6197-4dfc-9efb-80cc78fc958d" (UID: "75756d4a-6197-4dfc-9efb-80cc78fc958d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:58.414753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.414732 7010 scope.go:122] "RemoveContainer" containerID="7e1138c0afa7522f730c3f25e593181892c1c97b981d9ce65dee62f41a4bb060" Oct 30 13:38:58.414866 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.414845 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75756d4a-6197-4dfc-9efb-80cc78fc958d-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:58.415096 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:38:58.415078 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1138c0afa7522f730c3f25e593181892c1c97b981d9ce65dee62f41a4bb060\": container with ID starting with 7e1138c0afa7522f730c3f25e593181892c1c97b981d9ce65dee62f41a4bb060 not found: ID does not exist" containerID="7e1138c0afa7522f730c3f25e593181892c1c97b981d9ce65dee62f41a4bb060" Oct 30 13:38:58.415178 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.415119 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1138c0afa7522f730c3f25e593181892c1c97b981d9ce65dee62f41a4bb060"} err="failed to get container status \"7e1138c0afa7522f730c3f25e593181892c1c97b981d9ce65dee62f41a4bb060\": rpc error: code = NotFound desc = could not find container \"7e1138c0afa7522f730c3f25e593181892c1c97b981d9ce65dee62f41a4bb060\": container with ID starting with 7e1138c0afa7522f730c3f25e593181892c1c97b981d9ce65dee62f41a4bb060 not found: ID does not exist" Oct 30 13:38:58.416852 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.416829 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75756d4a-6197-4dfc-9efb-80cc78fc958d-kube-api-access-4h5zz" pod "75756d4a-6197-4dfc-9efb-80cc78fc958d" (UID: "75756d4a-6197-4dfc-9efb-80cc78fc958d"). InnerVolumeSpecName "kube-api-access-4h5zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:58.485508 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.481480 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj"] Oct 30 13:38:58.488843 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.486952 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b254d-6cc74c694c-gpqnj"] Oct 30 13:38:58.515696 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.515661 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a04bf8-d29c-425c-b351-61f785e861a3-kube-api-access-s4pc5\" (UniqueName: \"kubernetes.io/projected/75a04bf8-d29c-425c-b351-61f785e861a3-kube-api-access-s4pc5\") pod \"75a04bf8-d29c-425c-b351-61f785e861a3\" (UID: \"75a04bf8-d29c-425c-b351-61f785e861a3\") " Oct 30 13:38:58.515874 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.515816 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a04bf8-d29c-425c-b351-61f785e861a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a04bf8-d29c-425c-b351-61f785e861a3-plugins-config-volume\") pod \"75a04bf8-d29c-425c-b351-61f785e861a3\" (UID: \"75a04bf8-d29c-425c-b351-61f785e861a3\") " Oct 30 13:38:58.516225 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.516186 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a04bf8-d29c-425c-b351-61f785e861a3-plugins-config-volume" pod "75a04bf8-d29c-425c-b351-61f785e861a3" (UID: "75a04bf8-d29c-425c-b351-61f785e861a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:38:58.516559 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.516538 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h5zz\" (UniqueName: \"kubernetes.io/projected/75756d4a-6197-4dfc-9efb-80cc78fc958d-kube-api-access-4h5zz\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:58.516619 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.516565 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a04bf8-d29c-425c-b351-61f785e861a3-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:58.517960 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.517942 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a04bf8-d29c-425c-b351-61f785e861a3-kube-api-access-s4pc5" pod "75a04bf8-d29c-425c-b351-61f785e861a3" (UID: "75a04bf8-d29c-425c-b351-61f785e861a3"). InnerVolumeSpecName "kube-api-access-s4pc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:38:58.618020 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:58.617932 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4pc5\" (UniqueName: \"kubernetes.io/projected/75a04bf8-d29c-425c-b351-61f785e861a3-kube-api-access-s4pc5\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:38:59.404515 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:38:59.404464 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:38:59.411898 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:59.411869 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" Oct 30 13:38:59.412120 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:59.411902 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l" event={"ID":"75a04bf8-d29c-425c-b351-61f785e861a3","Type":"ContainerDied","Data":"a9d37a5e8acddac87707dd66205beb7bd89e345e89022926c1ff0ed657237c29"} Oct 30 13:38:59.412120 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:59.411980 7010 scope.go:122] "RemoveContainer" containerID="0d001311689fd034400db3c34f9046ae865f4020c153167c2629008f7284e8a6" Oct 30 13:38:59.412241 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:59.412131 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw" Oct 30 13:38:59.491368 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:59.491334 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f34026b-12db-4043-a58f-6d0344549ae7" path="/var/lib/kubelet/pods/5f34026b-12db-4043-a58f-6d0344549ae7/volumes" Oct 30 13:38:59.491788 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:59.491687 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l"] Oct 30 13:38:59.492433 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:59.492409 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b254d-6d64955646-5xd6l"] Oct 30 13:38:59.535262 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:59.535225 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw"] Oct 30 13:38:59.540212 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:38:59.540183 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b254d-588b769c95-j2jrw"] Oct 30 13:39:01.490797 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:01.490760 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75756d4a-6197-4dfc-9efb-80cc78fc958d" path="/var/lib/kubelet/pods/75756d4a-6197-4dfc-9efb-80cc78fc958d/volumes" Oct 30 13:39:01.491424 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:01.491402 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a04bf8-d29c-425c-b351-61f785e861a3" path="/var/lib/kubelet/pods/75a04bf8-d29c-425c-b351-61f785e861a3/volumes" Oct 30 13:39:04.271448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:04.271377 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" Oct 30 13:39:04.271842 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:04.271449 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" Oct 30 13:39:13.438452 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.438411 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6"] Oct 30 13:39:13.439710 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.439682 7010 cpu_manager.go:414] "containerMap: removing container" podUID="5f34026b-12db-4043-a58f-6d0344549ae7" containerName="maas-api" Oct 30 13:39:13.439710 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.439707 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f34026b-12db-4043-a58f-6d0344549ae7" containerName="maas-api" Oct 30 13:39:13.439882 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.439740 7010 cpu_manager.go:414] "containerMap: removing container" podUID="75756d4a-6197-4dfc-9efb-80cc78fc958d" containerName="payload-pre-processing" Oct 30 13:39:13.439882 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.439748 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75756d4a-6197-4dfc-9efb-80cc78fc958d" containerName="payload-pre-processing" Oct 30 13:39:13.439882 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.439823 7010 cpu_manager.go:414] "containerMap: removing container" podUID="75a04bf8-d29c-425c-b351-61f785e861a3" containerName="payload-processing" Oct 30 13:39:13.439882 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.439832 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a04bf8-d29c-425c-b351-61f785e861a3" containerName="payload-processing" Oct 30 13:39:13.440105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.439977 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="75756d4a-6197-4dfc-9efb-80cc78fc958d" containerName="payload-pre-processing" Oct 30 13:39:13.440105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.440007 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a04bf8-d29c-425c-b351-61f785e861a3" containerName="payload-processing" Oct 30 13:39:13.440105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.440019 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f34026b-12db-4043-a58f-6d0344549ae7" containerName="maas-api" Oct 30 13:39:13.445780 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.445755 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" Oct 30 13:39:13.463436 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.463401 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6"] Oct 30 13:39:13.472969 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.472936 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-78dc89\"" Oct 30 13:39:13.472969 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.472936 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-78dc89-dockercfg-gpwk5\"" Oct 30 13:39:13.555237 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.555204 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dc2205-c86e-4bd3-b364-15b5a3d6aee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6\" (UID: \"28dc2205-c86e-4bd3-b364-15b5a3d6aee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" Oct 30 13:39:13.555432 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.555376 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvfgx\" (UniqueName: \"kubernetes.io/projected/28dc2205-c86e-4bd3-b364-15b5a3d6aee4-kube-api-access-hvfgx\") pod \"payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6\" (UID: \"28dc2205-c86e-4bd3-b364-15b5a3d6aee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" Oct 30 13:39:13.586044 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.586009 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx"] Oct 30 13:39:13.590675 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.590648 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" Oct 30 13:39:13.610826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.610790 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx"] Oct 30 13:39:13.656788 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.656752 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvfgx\" (UniqueName: \"kubernetes.io/projected/28dc2205-c86e-4bd3-b364-15b5a3d6aee4-kube-api-access-hvfgx\") pod \"payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6\" (UID: \"28dc2205-c86e-4bd3-b364-15b5a3d6aee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" Oct 30 13:39:13.656983 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.656892 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvbdh\" (UniqueName: \"kubernetes.io/projected/36b712a0-c94f-4e1d-b769-56b308d4e7b1-kube-api-access-pvbdh\") pod \"payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx\" (UID: \"36b712a0-c94f-4e1d-b769-56b308d4e7b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" Oct 30 13:39:13.656983 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.656936 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dc2205-c86e-4bd3-b364-15b5a3d6aee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6\" (UID: \"28dc2205-c86e-4bd3-b364-15b5a3d6aee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" Oct 30 13:39:13.656983 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.656956 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b712a0-c94f-4e1d-b769-56b308d4e7b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx\" (UID: \"36b712a0-c94f-4e1d-b769-56b308d4e7b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" Oct 30 13:39:13.657781 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.657745 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dc2205-c86e-4bd3-b364-15b5a3d6aee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6\" (UID: \"28dc2205-c86e-4bd3-b364-15b5a3d6aee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" Oct 30 13:39:13.701678 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.698560 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvfgx\" (UniqueName: \"kubernetes.io/projected/28dc2205-c86e-4bd3-b364-15b5a3d6aee4-kube-api-access-hvfgx\") pod \"payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6\" (UID: \"28dc2205-c86e-4bd3-b364-15b5a3d6aee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" Oct 30 13:39:13.763895 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.762538 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvbdh\" (UniqueName: \"kubernetes.io/projected/36b712a0-c94f-4e1d-b769-56b308d4e7b1-kube-api-access-pvbdh\") pod \"payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx\" (UID: \"36b712a0-c94f-4e1d-b769-56b308d4e7b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" Oct 30 13:39:13.763895 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.762627 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b712a0-c94f-4e1d-b769-56b308d4e7b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx\" (UID: \"36b712a0-c94f-4e1d-b769-56b308d4e7b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" Oct 30 13:39:13.763895 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.763575 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b712a0-c94f-4e1d-b769-56b308d4e7b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx\" (UID: \"36b712a0-c94f-4e1d-b769-56b308d4e7b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" Oct 30 13:39:13.764661 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.764416 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" Oct 30 13:39:13.827432 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.827394 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvbdh\" (UniqueName: \"kubernetes.io/projected/36b712a0-c94f-4e1d-b769-56b308d4e7b1-kube-api-access-pvbdh\") pod \"payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx\" (UID: \"36b712a0-c94f-4e1d-b769-56b308d4e7b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" Oct 30 13:39:13.906566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:13.906512 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" Oct 30 13:39:14.180914 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:14.180882 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6"] Oct 30 13:39:14.182250 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:39:14.182226 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28dc2205_c86e_4bd3_b364_15b5a3d6aee4.slice/crio-986279fc018f01de2447216b81b860b04f83f7417ce10a0540e09149dffe9d04 WatchSource:0}: Error finding container 986279fc018f01de2447216b81b860b04f83f7417ce10a0540e09149dffe9d04: Status 404 returned error can't find the container with id 986279fc018f01de2447216b81b860b04f83f7417ce10a0540e09149dffe9d04 Oct 30 13:39:14.273743 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:14.273712 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx"] Oct 30 13:39:14.275176 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:39:14.275150 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b712a0_c94f_4e1d_b769_56b308d4e7b1.slice/crio-68dcfbfb748086e70ca853dc384733c78283608d3af932acd33ef4c81170f42c WatchSource:0}: Error finding container 68dcfbfb748086e70ca853dc384733c78283608d3af932acd33ef4c81170f42c: Status 404 returned error can't find the container with id 68dcfbfb748086e70ca853dc384733c78283608d3af932acd33ef4c81170f42c Oct 30 13:39:14.538563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:14.538410 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" event={"ID":"28dc2205-c86e-4bd3-b364-15b5a3d6aee4","Type":"ContainerStarted","Data":"fe6369a7f76008bf176028b11c1dc380090103006eb14882f40a13b52949794e"} Oct 30 13:39:14.538563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:14.538470 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" event={"ID":"28dc2205-c86e-4bd3-b364-15b5a3d6aee4","Type":"ContainerStarted","Data":"986279fc018f01de2447216b81b860b04f83f7417ce10a0540e09149dffe9d04"} Oct 30 13:39:14.538563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:14.538545 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" Oct 30 13:39:14.540620 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:14.540592 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" event={"ID":"36b712a0-c94f-4e1d-b769-56b308d4e7b1","Type":"ContainerStarted","Data":"03d057222714b25be11d47ac49b81bebec3df4a9f02406934804df533d35fa89"} Oct 30 13:39:14.540773 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:14.540642 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" event={"ID":"36b712a0-c94f-4e1d-b769-56b308d4e7b1","Type":"ContainerStarted","Data":"68dcfbfb748086e70ca853dc384733c78283608d3af932acd33ef4c81170f42c"} Oct 30 13:39:14.540773 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:14.540718 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" Oct 30 13:39:14.592317 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:14.592254 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" podStartSLOduration=1.5922391569999998 podStartE2EDuration="1.592239157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:14.590185447 +0000 UTC m=+2811.222372609" watchObservedRunningTime="2026-10-30 13:39:14.592239157 +0000 UTC m=+2811.224426318" Oct 30 13:39:14.633249 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:14.633191 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" podStartSLOduration=1.633177117 podStartE2EDuration="1.633177117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:14.632249895 +0000 UTC m=+2811.264437061" watchObservedRunningTime="2026-10-30 13:39:14.633177117 +0000 UTC m=+2811.265364279" Oct 30 13:39:25.550008 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:25.549976 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" Oct 30 13:39:25.550388 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:39:25.550051 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" Oct 30 13:39:29.402695 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:39:29.402650 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:39:59.400647 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:39:59.400608 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:40:25.392254 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:25.392221 7010 scope.go:122] "RemoveContainer" containerID="8ddd5e029357c181ffbd085859cd63cd77574ad8b7f50abd1e584880907d5cac" Oct 30 13:40:29.401051 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:40:29.401013 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:40:34.238769 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.238547 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2"] Oct 30 13:40:34.244541 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.244509 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" Oct 30 13:40:34.256854 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.256817 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2"] Oct 30 13:40:34.279889 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.279860 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c54f0f-dockercfg-dtxt7\"" Oct 30 13:40:34.280269 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.280237 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c54f0f\"" Oct 30 13:40:34.343741 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.343693 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d2121e-3ecd-4540-889b-3afd02a60f91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2\" (UID: \"84d2121e-3ecd-4540-889b-3afd02a60f91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" Oct 30 13:40:34.346581 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.346550 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmd87\" (UniqueName: \"kubernetes.io/projected/84d2121e-3ecd-4540-889b-3afd02a60f91-kube-api-access-gmd87\") pod \"payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2\" (UID: \"84d2121e-3ecd-4540-889b-3afd02a60f91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" Oct 30 13:40:34.379275 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.379242 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq"] Oct 30 13:40:34.385031 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.385007 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" Oct 30 13:40:34.398774 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.398747 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq"] Oct 30 13:40:34.448373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.448333 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpfst\" (UniqueName: \"kubernetes.io/projected/aa83aaa6-483b-4ae1-aaaa-2f30faf7f576-kube-api-access-rpfst\") pod \"payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq\" (UID: \"aa83aaa6-483b-4ae1-aaaa-2f30faf7f576\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" Oct 30 13:40:34.448603 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.448450 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d2121e-3ecd-4540-889b-3afd02a60f91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2\" (UID: \"84d2121e-3ecd-4540-889b-3afd02a60f91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" Oct 30 13:40:34.448686 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.448638 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmd87\" (UniqueName: \"kubernetes.io/projected/84d2121e-3ecd-4540-889b-3afd02a60f91-kube-api-access-gmd87\") pod \"payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2\" (UID: \"84d2121e-3ecd-4540-889b-3afd02a60f91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" Oct 30 13:40:34.448757 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.448701 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa83aaa6-483b-4ae1-aaaa-2f30faf7f576-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq\" (UID: \"aa83aaa6-483b-4ae1-aaaa-2f30faf7f576\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" Oct 30 13:40:34.449115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.449095 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d2121e-3ecd-4540-889b-3afd02a60f91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2\" (UID: \"84d2121e-3ecd-4540-889b-3afd02a60f91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" Oct 30 13:40:34.473004 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.472976 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmd87\" (UniqueName: \"kubernetes.io/projected/84d2121e-3ecd-4540-889b-3afd02a60f91-kube-api-access-gmd87\") pod \"payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2\" (UID: \"84d2121e-3ecd-4540-889b-3afd02a60f91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" Oct 30 13:40:34.550570 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.550450 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa83aaa6-483b-4ae1-aaaa-2f30faf7f576-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq\" (UID: \"aa83aaa6-483b-4ae1-aaaa-2f30faf7f576\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" Oct 30 13:40:34.550761 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.550663 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpfst\" (UniqueName: \"kubernetes.io/projected/aa83aaa6-483b-4ae1-aaaa-2f30faf7f576-kube-api-access-rpfst\") pod \"payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq\" (UID: \"aa83aaa6-483b-4ae1-aaaa-2f30faf7f576\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" Oct 30 13:40:34.551324 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.551301 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa83aaa6-483b-4ae1-aaaa-2f30faf7f576-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq\" (UID: \"aa83aaa6-483b-4ae1-aaaa-2f30faf7f576\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" Oct 30 13:40:34.574831 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.574798 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpfst\" (UniqueName: \"kubernetes.io/projected/aa83aaa6-483b-4ae1-aaaa-2f30faf7f576-kube-api-access-rpfst\") pod \"payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq\" (UID: \"aa83aaa6-483b-4ae1-aaaa-2f30faf7f576\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" Oct 30 13:40:34.627737 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.627705 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" Oct 30 13:40:34.706459 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.706425 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" Oct 30 13:40:34.941678 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:34.941652 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2"] Oct 30 13:40:34.943068 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:40:34.943038 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d2121e_3ecd_4540_889b_3afd02a60f91.slice/crio-752e6a402ba5677ce396c47f457a1bc0f2bf76c15dae2edefd4b89f50ab3f744 WatchSource:0}: Error finding container 752e6a402ba5677ce396c47f457a1bc0f2bf76c15dae2edefd4b89f50ab3f744: Status 404 returned error can't find the container with id 752e6a402ba5677ce396c47f457a1bc0f2bf76c15dae2edefd4b89f50ab3f744 Oct 30 13:40:35.043609 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:35.043572 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq"] Oct 30 13:40:35.044066 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:40:35.044046 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa83aaa6_483b_4ae1_aaaa_2f30faf7f576.slice/crio-aef5f1afd9bee875350276f34ed7bb62d248535c4ba9f4157131104d4880307a WatchSource:0}: Error finding container aef5f1afd9bee875350276f34ed7bb62d248535c4ba9f4157131104d4880307a: Status 404 returned error can't find the container with id aef5f1afd9bee875350276f34ed7bb62d248535c4ba9f4157131104d4880307a Oct 30 13:40:35.270855 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:35.270762 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" event={"ID":"84d2121e-3ecd-4540-889b-3afd02a60f91","Type":"ContainerStarted","Data":"56a0a01fd748b14d5167aba1fb4124dbde558e65e580633f41b584692fcffa09"} Oct 30 13:40:35.270855 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:35.270830 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" event={"ID":"84d2121e-3ecd-4540-889b-3afd02a60f91","Type":"ContainerStarted","Data":"752e6a402ba5677ce396c47f457a1bc0f2bf76c15dae2edefd4b89f50ab3f744"} Oct 30 13:40:35.271383 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:35.270863 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" Oct 30 13:40:35.272987 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:35.272959 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" event={"ID":"aa83aaa6-483b-4ae1-aaaa-2f30faf7f576","Type":"ContainerStarted","Data":"6397edabf904d17b04777c80a67a09b9e856e37e41170395028acca2d266d9df"} Oct 30 13:40:35.273409 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:35.273004 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" event={"ID":"aa83aaa6-483b-4ae1-aaaa-2f30faf7f576","Type":"ContainerStarted","Data":"aef5f1afd9bee875350276f34ed7bb62d248535c4ba9f4157131104d4880307a"} Oct 30 13:40:35.273409 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:35.273047 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" Oct 30 13:40:35.316774 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:35.316718 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" podStartSLOduration=1.316702892 podStartE2EDuration="1.316702892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:35.312877998 +0000 UTC m=+2891.945065163" watchObservedRunningTime="2026-10-30 13:40:35.316702892 +0000 UTC m=+2891.948890053" Oct 30 13:40:35.358459 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:35.358401 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" podStartSLOduration=1.358386885 podStartE2EDuration="1.358386885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:35.355468421 +0000 UTC m=+2891.987655574" watchObservedRunningTime="2026-10-30 13:40:35.358386885 +0000 UTC m=+2891.990574046" Oct 30 13:40:36.969985 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:36.969946 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp"] Oct 30 13:40:36.980780 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:36.980748 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" Oct 30 13:40:36.985066 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:36.985025 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp"] Oct 30 13:40:37.013142 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.011679 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-fca422\"" Oct 30 13:40:37.013142 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.011778 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-fca422-dockercfg-7jc2w\"" Oct 30 13:40:37.078384 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.078347 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7"] Oct 30 13:40:37.084183 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.084143 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" Oct 30 13:40:37.102994 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.102965 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7"] Oct 30 13:40:37.114429 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.114373 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ba48c8-ab14-4261-8529-eefa097249d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp\" (UID: \"c5ba48c8-ab14-4261-8529-eefa097249d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" Oct 30 13:40:37.114583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.114428 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87bf8\" (UniqueName: \"kubernetes.io/projected/c5ba48c8-ab14-4261-8529-eefa097249d4-kube-api-access-87bf8\") pod \"payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp\" (UID: \"c5ba48c8-ab14-4261-8529-eefa097249d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" Oct 30 13:40:37.114583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.114502 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zg7h\" (UniqueName: \"kubernetes.io/projected/57a206b4-1557-4b43-9b49-f49f981334e9-kube-api-access-4zg7h\") pod \"payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7\" (UID: \"57a206b4-1557-4b43-9b49-f49f981334e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" Oct 30 13:40:37.114780 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.114741 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a206b4-1557-4b43-9b49-f49f981334e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7\" (UID: \"57a206b4-1557-4b43-9b49-f49f981334e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" Oct 30 13:40:37.216587 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.216547 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a206b4-1557-4b43-9b49-f49f981334e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7\" (UID: \"57a206b4-1557-4b43-9b49-f49f981334e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" Oct 30 13:40:37.216771 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.216648 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ba48c8-ab14-4261-8529-eefa097249d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp\" (UID: \"c5ba48c8-ab14-4261-8529-eefa097249d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" Oct 30 13:40:37.216771 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.216675 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87bf8\" (UniqueName: \"kubernetes.io/projected/c5ba48c8-ab14-4261-8529-eefa097249d4-kube-api-access-87bf8\") pod \"payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp\" (UID: \"c5ba48c8-ab14-4261-8529-eefa097249d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" Oct 30 13:40:37.216771 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.216709 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zg7h\" (UniqueName: \"kubernetes.io/projected/57a206b4-1557-4b43-9b49-f49f981334e9-kube-api-access-4zg7h\") pod \"payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7\" (UID: \"57a206b4-1557-4b43-9b49-f49f981334e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" Oct 30 13:40:37.217395 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.217372 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a206b4-1557-4b43-9b49-f49f981334e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7\" (UID: \"57a206b4-1557-4b43-9b49-f49f981334e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" Oct 30 13:40:37.217615 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.217592 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ba48c8-ab14-4261-8529-eefa097249d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp\" (UID: \"c5ba48c8-ab14-4261-8529-eefa097249d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" Oct 30 13:40:37.241147 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.241072 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zg7h\" (UniqueName: \"kubernetes.io/projected/57a206b4-1557-4b43-9b49-f49f981334e9-kube-api-access-4zg7h\") pod \"payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7\" (UID: \"57a206b4-1557-4b43-9b49-f49f981334e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" Oct 30 13:40:37.241284 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.241185 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87bf8\" (UniqueName: \"kubernetes.io/projected/c5ba48c8-ab14-4261-8529-eefa097249d4-kube-api-access-87bf8\") pod \"payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp\" (UID: \"c5ba48c8-ab14-4261-8529-eefa097249d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" Oct 30 13:40:37.319548 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.319508 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" Oct 30 13:40:37.423022 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.422982 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" Oct 30 13:40:37.640283 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.640256 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp"] Oct 30 13:40:37.641076 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:40:37.641048 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ba48c8_ab14_4261_8529_eefa097249d4.slice/crio-d6b79723435f0f3168228c1973fa1d0d920d3ee1525a1ccc3e1202fcb9810971 WatchSource:0}: Error finding container d6b79723435f0f3168228c1973fa1d0d920d3ee1525a1ccc3e1202fcb9810971: Status 404 returned error can't find the container with id d6b79723435f0f3168228c1973fa1d0d920d3ee1525a1ccc3e1202fcb9810971 Oct 30 13:40:37.753152 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:37.753119 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7"] Oct 30 13:40:37.755740 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:40:37.755701 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a206b4_1557_4b43_9b49_f49f981334e9.slice/crio-119d633ba069e55bbcb66276bacffb4bf21c36e91766e0369db7bcd45da49475 WatchSource:0}: Error finding container 119d633ba069e55bbcb66276bacffb4bf21c36e91766e0369db7bcd45da49475: Status 404 returned error can't find the container with id 119d633ba069e55bbcb66276bacffb4bf21c36e91766e0369db7bcd45da49475 Oct 30 13:40:38.298264 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:38.298228 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" event={"ID":"c5ba48c8-ab14-4261-8529-eefa097249d4","Type":"ContainerStarted","Data":"2969761ec7a439b998e8833e48c7fd899253dfc11ca8e475414df2427130b1c5"} Oct 30 13:40:38.298826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:38.298296 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" Oct 30 13:40:38.298826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:38.298318 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" event={"ID":"c5ba48c8-ab14-4261-8529-eefa097249d4","Type":"ContainerStarted","Data":"d6b79723435f0f3168228c1973fa1d0d920d3ee1525a1ccc3e1202fcb9810971"} Oct 30 13:40:38.300028 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:38.299998 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" event={"ID":"57a206b4-1557-4b43-9b49-f49f981334e9","Type":"ContainerStarted","Data":"876cd05627f298b22e4f62dceecfae3f747f4cb0e435f32f4eb4c91e9d9f6d93"} Oct 30 13:40:38.300143 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:38.300052 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" event={"ID":"57a206b4-1557-4b43-9b49-f49f981334e9","Type":"ContainerStarted","Data":"119d633ba069e55bbcb66276bacffb4bf21c36e91766e0369db7bcd45da49475"} Oct 30 13:40:38.300143 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:38.300081 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" Oct 30 13:40:38.345199 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:38.345144 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" podStartSLOduration=2.345131613 podStartE2EDuration="2.345131613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:38.342001152 +0000 UTC m=+2894.974188313" watchObservedRunningTime="2026-10-30 13:40:38.345131613 +0000 UTC m=+2894.977318772" Oct 30 13:40:38.385013 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:38.384952 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" podStartSLOduration=1.384871963 podStartE2EDuration="1.384871963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:38.384336433 +0000 UTC m=+2895.016523602" watchObservedRunningTime="2026-10-30 13:40:38.384871963 +0000 UTC m=+2895.017059126" Oct 30 13:40:46.081775 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:46.081736 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt"] Oct 30 13:40:46.082242 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:46.081990 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" podUID="3f46ced9-2698-442f-8e00-1f7766b92d3c" containerName="istio-proxy" containerID="cri-o://82dc6d099214a6d41c42568e16a51249f70d1007809b594895cfbf09dc7e1518" gracePeriod=30 Oct 30 13:40:46.283260 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:46.283224 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" Oct 30 13:40:46.283440 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:46.283326 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" Oct 30 13:40:49.309344 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:49.309309 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" Oct 30 13:40:49.309872 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:49.309730 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" Oct 30 13:40:51.387687 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.387657 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:40:51.442652 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.442610 7010 generic.go:356] "Generic (PLEG): container finished" podID="3f46ced9-2698-442f-8e00-1f7766b92d3c" containerID="82dc6d099214a6d41c42568e16a51249f70d1007809b594895cfbf09dc7e1518" exitCode=0 Oct 30 13:40:51.442826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.442664 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" Oct 30 13:40:51.442826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.442691 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" event={"ID":"3f46ced9-2698-442f-8e00-1f7766b92d3c","Type":"ContainerDied","Data":"82dc6d099214a6d41c42568e16a51249f70d1007809b594895cfbf09dc7e1518"} Oct 30 13:40:51.442826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.442756 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt" event={"ID":"3f46ced9-2698-442f-8e00-1f7766b92d3c","Type":"ContainerDied","Data":"8cbd0148ecdf30e4803108c42001851baf0d2473d63ab191779206c92e9e3168"} Oct 30 13:40:51.442826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.442785 7010 scope.go:122] "RemoveContainer" containerID="82dc6d099214a6d41c42568e16a51249f70d1007809b594895cfbf09dc7e1518" Oct 30 13:40:51.455884 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.455855 7010 scope.go:122] "RemoveContainer" containerID="82dc6d099214a6d41c42568e16a51249f70d1007809b594895cfbf09dc7e1518" Oct 30 13:40:51.456294 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:40:51.456270 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82dc6d099214a6d41c42568e16a51249f70d1007809b594895cfbf09dc7e1518\": container with ID starting with 82dc6d099214a6d41c42568e16a51249f70d1007809b594895cfbf09dc7e1518 not found: ID does not exist" containerID="82dc6d099214a6d41c42568e16a51249f70d1007809b594895cfbf09dc7e1518" Oct 30 13:40:51.456404 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.456307 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82dc6d099214a6d41c42568e16a51249f70d1007809b594895cfbf09dc7e1518"} err="failed to get container status \"82dc6d099214a6d41c42568e16a51249f70d1007809b594895cfbf09dc7e1518\": rpc error: code = NotFound desc = could not find container \"82dc6d099214a6d41c42568e16a51249f70d1007809b594895cfbf09dc7e1518\": container with ID starting with 82dc6d099214a6d41c42568e16a51249f70d1007809b594895cfbf09dc7e1518 not found: ID does not exist" Oct 30 13:40:51.501369 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.501336 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-workload-certs\") pod \"3f46ced9-2698-442f-8e00-1f7766b92d3c\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " Oct 30 13:40:51.501592 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.501398 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-credential-socket\") pod \"3f46ced9-2698-442f-8e00-1f7766b92d3c\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " Oct 30 13:40:51.501592 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.501542 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f46ced9-2698-442f-8e00-1f7766b92d3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f46ced9-2698-442f-8e00-1f7766b92d3c-istiod-ca-cert\") pod \"3f46ced9-2698-442f-8e00-1f7766b92d3c\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " Oct 30 13:40:51.501732 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.501605 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-envoy\") pod \"3f46ced9-2698-442f-8e00-1f7766b92d3c\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " Oct 30 13:40:51.501732 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.501653 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-credential-socket" pod "3f46ced9-2698-442f-8e00-1f7766b92d3c" (UID: "3f46ced9-2698-442f-8e00-1f7766b92d3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:51.501732 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.501670 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-data\") pod \"3f46ced9-2698-442f-8e00-1f7766b92d3c\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " Oct 30 13:40:51.501732 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.501681 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-workload-certs" pod "3f46ced9-2698-442f-8e00-1f7766b92d3c" (UID: "3f46ced9-2698-442f-8e00-1f7766b92d3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:51.501954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.501739 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f46ced9-2698-442f-8e00-1f7766b92d3c-kube-api-access-85ml5\" (UniqueName: \"kubernetes.io/projected/3f46ced9-2698-442f-8e00-1f7766b92d3c-kube-api-access-85ml5\") pod \"3f46ced9-2698-442f-8e00-1f7766b92d3c\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " Oct 30 13:40:51.501954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.501773 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-workload-socket\") pod \"3f46ced9-2698-442f-8e00-1f7766b92d3c\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " Oct 30 13:40:51.501954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.501819 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-token\" (UniqueName: \"kubernetes.io/projected/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-token\") pod \"3f46ced9-2698-442f-8e00-1f7766b92d3c\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " Oct 30 13:40:51.501954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.501909 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-data" pod "3f46ced9-2698-442f-8e00-1f7766b92d3c" (UID: "3f46ced9-2698-442f-8e00-1f7766b92d3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:51.502143 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.501973 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-podinfo\") pod \"3f46ced9-2698-442f-8e00-1f7766b92d3c\" (UID: \"3f46ced9-2698-442f-8e00-1f7766b92d3c\") " Oct 30 13:40:51.502143 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.502039 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-workload-socket" pod "3f46ced9-2698-442f-8e00-1f7766b92d3c" (UID: "3f46ced9-2698-442f-8e00-1f7766b92d3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:51.502143 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.502073 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f46ced9-2698-442f-8e00-1f7766b92d3c-istiod-ca-cert" pod "3f46ced9-2698-442f-8e00-1f7766b92d3c" (UID: "3f46ced9-2698-442f-8e00-1f7766b92d3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:40:51.502569 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.502545 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.502569 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.502571 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.502769 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.502585 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f46ced9-2698-442f-8e00-1f7766b92d3c-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.502769 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.502601 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.502769 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.502613 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.504532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.504068 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-envoy" pod "3f46ced9-2698-442f-8e00-1f7766b92d3c" (UID: "3f46ced9-2698-442f-8e00-1f7766b92d3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:40:51.504532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.504162 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-podinfo" pod "3f46ced9-2698-442f-8e00-1f7766b92d3c" (UID: "3f46ced9-2698-442f-8e00-1f7766b92d3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:40:51.504532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.504195 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-token" pod "3f46ced9-2698-442f-8e00-1f7766b92d3c" (UID: "3f46ced9-2698-442f-8e00-1f7766b92d3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:51.504532 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.504320 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f46ced9-2698-442f-8e00-1f7766b92d3c-kube-api-access-85ml5" pod "3f46ced9-2698-442f-8e00-1f7766b92d3c" (UID: "3f46ced9-2698-442f-8e00-1f7766b92d3c"). InnerVolumeSpecName "kube-api-access-85ml5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:40:51.604206 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.604171 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.604206 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.604205 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85ml5\" (UniqueName: \"kubernetes.io/projected/3f46ced9-2698-442f-8e00-1f7766b92d3c-kube-api-access-85ml5\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.604206 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.604216 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.604525 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.604227 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f46ced9-2698-442f-8e00-1f7766b92d3c-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:40:51.792199 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.792152 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt"] Oct 30 13:40:51.799667 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:51.799627 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4b0ea9-openshift-default-76459f746f-mrtzt"] Oct 30 13:40:53.490633 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:53.490595 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f46ced9-2698-442f-8e00-1f7766b92d3c" path="/var/lib/kubelet/pods/3f46ced9-2698-442f-8e00-1f7766b92d3c/volumes" Oct 30 13:40:57.670247 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.670201 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb"] Oct 30 13:40:57.671015 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.670997 7010 cpu_manager.go:414] "containerMap: removing container" podUID="3f46ced9-2698-442f-8e00-1f7766b92d3c" containerName="istio-proxy" Oct 30 13:40:57.671063 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.671017 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f46ced9-2698-442f-8e00-1f7766b92d3c" containerName="istio-proxy" Oct 30 13:40:57.671159 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.671149 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f46ced9-2698-442f-8e00-1f7766b92d3c" containerName="istio-proxy" Oct 30 13:40:57.674675 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.674658 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.710007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.709824 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb"] Oct 30 13:40:57.779926 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.779886 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8ee35e-openshift-default-dockercfg-27hz2\"" Oct 30 13:40:57.809328 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.809291 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cnqs\" (UniqueName: \"kubernetes.io/projected/9e028b61-6039-4f07-b7a9-082d7b51c633-kube-api-access-6cnqs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.809551 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.809343 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.809551 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.809509 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e028b61-6039-4f07-b7a9-082d7b51c633-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.809707 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.809647 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.809773 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.809718 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.809880 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.809864 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.809946 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.809931 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.810003 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.809976 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.810088 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.810074 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.911461 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.911420 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e028b61-6039-4f07-b7a9-082d7b51c633-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.911712 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.911515 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.911712 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.911554 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.911712 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.911656 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.911900 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.911734 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.911900 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.911775 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.911900 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.911823 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.911900 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.911865 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cnqs\" (UniqueName: \"kubernetes.io/projected/9e028b61-6039-4f07-b7a9-082d7b51c633-kube-api-access-6cnqs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.912113 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.911914 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.912237 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.912213 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.912421 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.912396 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e028b61-6039-4f07-b7a9-082d7b51c633-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.912580 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.912559 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.912656 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.912578 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.912722 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.912693 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.914533 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.914509 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.914772 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.914755 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.935654 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.935577 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.936002 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.935980 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cnqs\" (UniqueName: \"kubernetes.io/projected/9e028b61-6039-4f07-b7a9-082d7b51c633-kube-api-access-6cnqs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:57.992219 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:57.992183 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:58.328130 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:58.328101 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb"] Oct 30 13:40:58.330895 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:40:58.330865 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e028b61_6039_4f07_b7a9_082d7b51c633.slice/crio-784a1beaa9255fec605e6ad6cabff0e62d6521ef21e06ad50d4afcd409348a34 WatchSource:0}: Error finding container 784a1beaa9255fec605e6ad6cabff0e62d6521ef21e06ad50d4afcd409348a34: Status 404 returned error can't find the container with id 784a1beaa9255fec605e6ad6cabff0e62d6521ef21e06ad50d4afcd409348a34 Oct 30 13:40:58.334025 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:58.333982 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:40:58.334126 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:58.334081 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:40:58.334164 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:58.334123 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:40:58.494339 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:58.494297 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" event={"ID":"9e028b61-6039-4f07-b7a9-082d7b51c633","Type":"ContainerStarted","Data":"a683036c84a1166248026ae873ee53aa97c1355e9f7005a3814359a8045fa4b5"} Oct 30 13:40:58.494576 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:58.494364 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" event={"ID":"9e028b61-6039-4f07-b7a9-082d7b51c633","Type":"ContainerStarted","Data":"784a1beaa9255fec605e6ad6cabff0e62d6521ef21e06ad50d4afcd409348a34"} Oct 30 13:40:58.543726 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:58.543665 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" podStartSLOduration=1.543645196 podStartE2EDuration="1.543645196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:58.541318326 +0000 UTC m=+2915.173505494" watchObservedRunningTime="2026-10-30 13:40:58.543645196 +0000 UTC m=+2915.175832359" Oct 30 13:40:58.992382 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:40:58.992343 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:40:59.425118 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:40:59.425077 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:41:00.023507 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:00.013127 7010 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 13:41:00.024947 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:00.024911 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" podUID="9e028b61-6039-4f07-b7a9-082d7b51c633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 13:41:00.997437 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:00.996608 7010 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 13:41:00.997437 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:00.996676 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" podUID="9e028b61-6039-4f07-b7a9-082d7b51c633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 13:41:01.528609 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:01.528572 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz"] Oct 30 13:41:02.007514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:02.006681 7010 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 13:41:02.007514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:02.006745 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" podUID="9e028b61-6039-4f07-b7a9-082d7b51c633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 13:41:03.008673 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:03.008630 7010 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 13:41:03.009232 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:03.008697 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" podUID="9e028b61-6039-4f07-b7a9-082d7b51c633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 13:41:03.994389 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:03.994346 7010 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 13:41:03.994743 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:03.994705 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" podUID="9e028b61-6039-4f07-b7a9-082d7b51c633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 13:41:05.000725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:05.000693 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:41:05.001172 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:05.001147 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:41:05.002408 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:05.002384 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:41:07.768959 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:07.768918 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr"] Oct 30 13:41:07.769442 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:07.769208 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" podUID="e943dcc3-df3a-45aa-ab0c-07588840101b" containerName="payload-processing" containerID="cri-o://a07795828bfe325cf3916b06288c63960a7e8f7ac57724ca8051867104e95606" gracePeriod=30 Oct 30 13:41:07.804120 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:07.804080 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp"] Oct 30 13:41:07.804636 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:07.804575 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" podUID="069c8497-8a23-4175-b162-0e0adb63a7f3" containerName="payload-pre-processing" containerID="cri-o://801c8aa3c21eb2ca562d66a810b309725d48620793aa53f0ab8ca86459e3b2c7" gracePeriod=30 Oct 30 13:41:08.332967 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.332940 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" Oct 30 13:41:08.336892 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.336867 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" Oct 30 13:41:08.461015 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.460967 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e943dcc3-df3a-45aa-ab0c-07588840101b-kube-api-access-rwb8n\" (UniqueName: \"kubernetes.io/projected/e943dcc3-df3a-45aa-ab0c-07588840101b-kube-api-access-rwb8n\") pod \"e943dcc3-df3a-45aa-ab0c-07588840101b\" (UID: \"e943dcc3-df3a-45aa-ab0c-07588840101b\") " Oct 30 13:41:08.461240 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.461127 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/069c8497-8a23-4175-b162-0e0adb63a7f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069c8497-8a23-4175-b162-0e0adb63a7f3-plugins-config-volume\") pod \"069c8497-8a23-4175-b162-0e0adb63a7f3\" (UID: \"069c8497-8a23-4175-b162-0e0adb63a7f3\") " Oct 30 13:41:08.461240 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.461222 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069c8497-8a23-4175-b162-0e0adb63a7f3-kube-api-access-xvnf4\" (UniqueName: \"kubernetes.io/projected/069c8497-8a23-4175-b162-0e0adb63a7f3-kube-api-access-xvnf4\") pod \"069c8497-8a23-4175-b162-0e0adb63a7f3\" (UID: \"069c8497-8a23-4175-b162-0e0adb63a7f3\") " Oct 30 13:41:08.461375 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.461312 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e943dcc3-df3a-45aa-ab0c-07588840101b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e943dcc3-df3a-45aa-ab0c-07588840101b-plugins-config-volume\") pod \"e943dcc3-df3a-45aa-ab0c-07588840101b\" (UID: \"e943dcc3-df3a-45aa-ab0c-07588840101b\") " Oct 30 13:41:08.461945 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.461914 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/069c8497-8a23-4175-b162-0e0adb63a7f3-plugins-config-volume" pod "069c8497-8a23-4175-b162-0e0adb63a7f3" (UID: "069c8497-8a23-4175-b162-0e0adb63a7f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:08.462118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.462003 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e943dcc3-df3a-45aa-ab0c-07588840101b-plugins-config-volume" pod "e943dcc3-df3a-45aa-ab0c-07588840101b" (UID: "e943dcc3-df3a-45aa-ab0c-07588840101b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:08.462387 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.462365 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069c8497-8a23-4175-b162-0e0adb63a7f3-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:08.462538 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.462521 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e943dcc3-df3a-45aa-ab0c-07588840101b-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:08.463988 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.463960 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069c8497-8a23-4175-b162-0e0adb63a7f3-kube-api-access-xvnf4" pod "069c8497-8a23-4175-b162-0e0adb63a7f3" (UID: "069c8497-8a23-4175-b162-0e0adb63a7f3"). InnerVolumeSpecName "kube-api-access-xvnf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:08.464226 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.464203 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e943dcc3-df3a-45aa-ab0c-07588840101b-kube-api-access-rwb8n" pod "e943dcc3-df3a-45aa-ab0c-07588840101b" (UID: "e943dcc3-df3a-45aa-ab0c-07588840101b"). InnerVolumeSpecName "kube-api-access-rwb8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:08.564649 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.564612 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwb8n\" (UniqueName: \"kubernetes.io/projected/e943dcc3-df3a-45aa-ab0c-07588840101b-kube-api-access-rwb8n\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:08.564649 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.564644 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvnf4\" (UniqueName: \"kubernetes.io/projected/069c8497-8a23-4175-b162-0e0adb63a7f3-kube-api-access-xvnf4\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:08.589714 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.589680 7010 generic.go:356] "Generic (PLEG): container finished" podID="e943dcc3-df3a-45aa-ab0c-07588840101b" containerID="a07795828bfe325cf3916b06288c63960a7e8f7ac57724ca8051867104e95606" exitCode=0 Oct 30 13:41:08.589918 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.589731 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" Oct 30 13:41:08.589918 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.589794 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" event={"ID":"e943dcc3-df3a-45aa-ab0c-07588840101b","Type":"ContainerDied","Data":"a07795828bfe325cf3916b06288c63960a7e8f7ac57724ca8051867104e95606"} Oct 30 13:41:08.589918 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.589843 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr" event={"ID":"e943dcc3-df3a-45aa-ab0c-07588840101b","Type":"ContainerDied","Data":"6c9d27a1f05159d5bf64d232cd80409533e4df1a190cd125a2de6e2552ab0c95"} Oct 30 13:41:08.589918 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.589871 7010 scope.go:122] "RemoveContainer" containerID="a07795828bfe325cf3916b06288c63960a7e8f7ac57724ca8051867104e95606" Oct 30 13:41:08.591566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.591534 7010 generic.go:356] "Generic (PLEG): container finished" podID="069c8497-8a23-4175-b162-0e0adb63a7f3" containerID="801c8aa3c21eb2ca562d66a810b309725d48620793aa53f0ab8ca86459e3b2c7" exitCode=0 Oct 30 13:41:08.591683 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.591571 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" event={"ID":"069c8497-8a23-4175-b162-0e0adb63a7f3","Type":"ContainerDied","Data":"801c8aa3c21eb2ca562d66a810b309725d48620793aa53f0ab8ca86459e3b2c7"} Oct 30 13:41:08.591683 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.591584 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" Oct 30 13:41:08.591683 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.591614 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp" event={"ID":"069c8497-8a23-4175-b162-0e0adb63a7f3","Type":"ContainerDied","Data":"861c14b6f48ea92afb7268c1953290d989b2916360dfa75bc3ae718c704aa64f"} Oct 30 13:41:08.605809 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.605781 7010 scope.go:122] "RemoveContainer" containerID="a07795828bfe325cf3916b06288c63960a7e8f7ac57724ca8051867104e95606" Oct 30 13:41:08.606219 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:41:08.606189 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07795828bfe325cf3916b06288c63960a7e8f7ac57724ca8051867104e95606\": container with ID starting with a07795828bfe325cf3916b06288c63960a7e8f7ac57724ca8051867104e95606 not found: ID does not exist" containerID="a07795828bfe325cf3916b06288c63960a7e8f7ac57724ca8051867104e95606" Oct 30 13:41:08.606323 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.606231 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07795828bfe325cf3916b06288c63960a7e8f7ac57724ca8051867104e95606"} err="failed to get container status \"a07795828bfe325cf3916b06288c63960a7e8f7ac57724ca8051867104e95606\": rpc error: code = NotFound desc = could not find container \"a07795828bfe325cf3916b06288c63960a7e8f7ac57724ca8051867104e95606\": container with ID starting with a07795828bfe325cf3916b06288c63960a7e8f7ac57724ca8051867104e95606 not found: ID does not exist" Oct 30 13:41:08.606323 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.606257 7010 scope.go:122] "RemoveContainer" containerID="801c8aa3c21eb2ca562d66a810b309725d48620793aa53f0ab8ca86459e3b2c7" Oct 30 13:41:08.616542 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.616508 7010 scope.go:122] "RemoveContainer" containerID="801c8aa3c21eb2ca562d66a810b309725d48620793aa53f0ab8ca86459e3b2c7" Oct 30 13:41:08.616869 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:41:08.616848 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"801c8aa3c21eb2ca562d66a810b309725d48620793aa53f0ab8ca86459e3b2c7\": container with ID starting with 801c8aa3c21eb2ca562d66a810b309725d48620793aa53f0ab8ca86459e3b2c7 not found: ID does not exist" containerID="801c8aa3c21eb2ca562d66a810b309725d48620793aa53f0ab8ca86459e3b2c7" Oct 30 13:41:08.616937 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.616880 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"801c8aa3c21eb2ca562d66a810b309725d48620793aa53f0ab8ca86459e3b2c7"} err="failed to get container status \"801c8aa3c21eb2ca562d66a810b309725d48620793aa53f0ab8ca86459e3b2c7\": rpc error: code = NotFound desc = could not find container \"801c8aa3c21eb2ca562d66a810b309725d48620793aa53f0ab8ca86459e3b2c7\": container with ID starting with 801c8aa3c21eb2ca562d66a810b309725d48620793aa53f0ab8ca86459e3b2c7 not found: ID does not exist" Oct 30 13:41:08.649570 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.649529 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp"] Oct 30 13:41:08.652408 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.652382 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-428a0e-7578d589d7-74hqp"] Oct 30 13:41:08.721359 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.721242 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr"] Oct 30 13:41:08.727611 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:08.727472 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-428a0e-6c578c9ffd-d26vr"] Oct 30 13:41:09.510503 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:09.510451 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="069c8497-8a23-4175-b162-0e0adb63a7f3" path="/var/lib/kubelet/pods/069c8497-8a23-4175-b162-0e0adb63a7f3/volumes" Oct 30 13:41:09.511112 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:09.511074 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e943dcc3-df3a-45aa-ab0c-07588840101b" path="/var/lib/kubelet/pods/e943dcc3-df3a-45aa-ab0c-07588840101b/volumes" Oct 30 13:41:16.551316 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:16.551242 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" containerID="cri-o://153a75c9201cafe5c1f9951efd2043df419b0080154f7c062ed91e31f35a7dc7" gracePeriod=45 Oct 30 13:41:17.000433 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.000402 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:41:17.031616 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.031579 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-dshm\") pod \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " Oct 30 13:41:17.031832 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.031672 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f9a281c-7ca6-4157-9f9a-e86363233dcb-kube-api-access-752v8\" (UniqueName: \"kubernetes.io/projected/7f9a281c-7ca6-4157-9f9a-e86363233dcb-kube-api-access-752v8\") pod \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " Oct 30 13:41:17.031832 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.031769 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f9a281c-7ca6-4157-9f9a-e86363233dcb-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f9a281c-7ca6-4157-9f9a-e86363233dcb-tls-certs\") pod \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " Oct 30 13:41:17.031832 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.031816 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-home\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-home\") pod \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " Oct 30 13:41:17.032022 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.031913 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-tmp-dir\") pod \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " Oct 30 13:41:17.032022 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.031981 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-model-cache\") pod \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\" (UID: \"7f9a281c-7ca6-4157-9f9a-e86363233dcb\") " Oct 30 13:41:17.032194 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.032164 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-home" pod "7f9a281c-7ca6-4157-9f9a-e86363233dcb" (UID: "7f9a281c-7ca6-4157-9f9a-e86363233dcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:17.032266 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.032174 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-tmp-dir" pod "7f9a281c-7ca6-4157-9f9a-e86363233dcb" (UID: "7f9a281c-7ca6-4157-9f9a-e86363233dcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:17.032397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.032360 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-model-cache" pod "7f9a281c-7ca6-4157-9f9a-e86363233dcb" (UID: "7f9a281c-7ca6-4157-9f9a-e86363233dcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:17.032664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.032643 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:17.032753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.032670 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:17.032753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.032711 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:17.034060 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.034032 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f9a281c-7ca6-4157-9f9a-e86363233dcb-tls-certs" pod "7f9a281c-7ca6-4157-9f9a-e86363233dcb" (UID: "7f9a281c-7ca6-4157-9f9a-e86363233dcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:41:17.034150 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.034038 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-dshm" pod "7f9a281c-7ca6-4157-9f9a-e86363233dcb" (UID: "7f9a281c-7ca6-4157-9f9a-e86363233dcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:17.034150 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.034116 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9a281c-7ca6-4157-9f9a-e86363233dcb-kube-api-access-752v8" pod "7f9a281c-7ca6-4157-9f9a-e86363233dcb" (UID: "7f9a281c-7ca6-4157-9f9a-e86363233dcb"). InnerVolumeSpecName "kube-api-access-752v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:17.133855 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.133814 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f9a281c-7ca6-4157-9f9a-e86363233dcb-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:17.133855 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.133848 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-752v8\" (UniqueName: \"kubernetes.io/projected/7f9a281c-7ca6-4157-9f9a-e86363233dcb-kube-api-access-752v8\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:17.133855 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.133859 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f9a281c-7ca6-4157-9f9a-e86363233dcb-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:17.743046 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.743009 7010 generic.go:356] "Generic (PLEG): container finished" podID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerID="153a75c9201cafe5c1f9951efd2043df419b0080154f7c062ed91e31f35a7dc7" exitCode=0 Oct 30 13:41:17.743563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.743054 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" event={"ID":"7f9a281c-7ca6-4157-9f9a-e86363233dcb","Type":"ContainerDied","Data":"153a75c9201cafe5c1f9951efd2043df419b0080154f7c062ed91e31f35a7dc7"} Oct 30 13:41:17.743563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.743079 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" Oct 30 13:41:17.743563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.743090 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" event={"ID":"7f9a281c-7ca6-4157-9f9a-e86363233dcb","Type":"ContainerDied","Data":"a33c93b8d9343a123eb853892d975de724558cad202fe90452ce0be633c491f2"} Oct 30 13:41:17.743563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.743113 7010 scope.go:122] "RemoveContainer" containerID="153a75c9201cafe5c1f9951efd2043df419b0080154f7c062ed91e31f35a7dc7" Oct 30 13:41:17.753805 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.753773 7010 scope.go:122] "RemoveContainer" containerID="153a75c9201cafe5c1f9951efd2043df419b0080154f7c062ed91e31f35a7dc7" Oct 30 13:41:17.754184 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:41:17.754162 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153a75c9201cafe5c1f9951efd2043df419b0080154f7c062ed91e31f35a7dc7\": container with ID starting with 153a75c9201cafe5c1f9951efd2043df419b0080154f7c062ed91e31f35a7dc7 not found: ID does not exist" containerID="153a75c9201cafe5c1f9951efd2043df419b0080154f7c062ed91e31f35a7dc7" Oct 30 13:41:17.754273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.754200 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153a75c9201cafe5c1f9951efd2043df419b0080154f7c062ed91e31f35a7dc7"} err="failed to get container status \"153a75c9201cafe5c1f9951efd2043df419b0080154f7c062ed91e31f35a7dc7\": rpc error: code = NotFound desc = could not find container \"153a75c9201cafe5c1f9951efd2043df419b0080154f7c062ed91e31f35a7dc7\": container with ID starting with 153a75c9201cafe5c1f9951efd2043df419b0080154f7c062ed91e31f35a7dc7 not found: ID does not exist" Oct 30 13:41:17.797629 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.797591 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz"] Oct 30 13:41:17.810741 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.810708 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz"] Oct 30 13:41:17.969974 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:17.969927 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cb58b8/premium-w2-cb58b8-kserve-76f66dfd4b-dnnjz" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 13:41:19.499129 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:19.499089 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" path="/var/lib/kubelet/pods/7f9a281c-7ca6-4157-9f9a-e86363233dcb/volumes" Oct 30 13:41:29.409120 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:41:29.409075 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:41:30.201966 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.201926 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj"] Oct 30 13:41:30.202727 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.202703 7010 cpu_manager.go:414] "containerMap: removing container" podUID="e943dcc3-df3a-45aa-ab0c-07588840101b" containerName="payload-processing" Oct 30 13:41:30.202727 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.202726 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e943dcc3-df3a-45aa-ab0c-07588840101b" containerName="payload-processing" Oct 30 13:41:30.202903 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.202769 7010 cpu_manager.go:414] "containerMap: removing container" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" Oct 30 13:41:30.202903 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.202777 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" Oct 30 13:41:30.202903 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.202792 7010 cpu_manager.go:414] "containerMap: removing container" podUID="069c8497-8a23-4175-b162-0e0adb63a7f3" containerName="payload-pre-processing" Oct 30 13:41:30.202903 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.202801 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="069c8497-8a23-4175-b162-0e0adb63a7f3" containerName="payload-pre-processing" Oct 30 13:41:30.203109 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.203003 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f9a281c-7ca6-4157-9f9a-e86363233dcb" containerName="main" Oct 30 13:41:30.203109 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.203023 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="069c8497-8a23-4175-b162-0e0adb63a7f3" containerName="payload-pre-processing" Oct 30 13:41:30.203109 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.203045 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="e943dcc3-df3a-45aa-ab0c-07588840101b" containerName="payload-processing" Oct 30 13:41:30.213173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.213134 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.230536 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.230499 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj"] Oct 30 13:41:30.239359 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.239320 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-cb58b8-openshift-default-dockercfg-dftld\"" Oct 30 13:41:30.261996 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.261955 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.262180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.262066 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.262180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.262101 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4764645c-c41c-4085-8131-424e1ff46c72-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.262180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.262135 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.262304 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.262277 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4764645c-c41c-4085-8131-424e1ff46c72-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.262475 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.262440 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.262582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.262542 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.262649 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.262583 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4764645c-c41c-4085-8131-424e1ff46c72-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.262710 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.262653 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swglv\" (UniqueName: \"kubernetes.io/projected/4764645c-c41c-4085-8131-424e1ff46c72-kube-api-access-swglv\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.364463 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.364426 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.364713 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.364484 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4764645c-c41c-4085-8131-424e1ff46c72-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.364713 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.364605 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swglv\" (UniqueName: \"kubernetes.io/projected/4764645c-c41c-4085-8131-424e1ff46c72-kube-api-access-swglv\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.364713 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.364626 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.364713 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.364671 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.364962 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.364795 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4764645c-c41c-4085-8131-424e1ff46c72-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.364962 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.364887 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.364962 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.364948 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4764645c-c41c-4085-8131-424e1ff46c72-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.365137 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.365075 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.365137 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.365087 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-credential-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.365247 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.365174 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-workload-certs\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.365357 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.365330 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-istio-data\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.365438 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.365416 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4764645c-c41c-4085-8131-424e1ff46c72-istiod-ca-cert\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.365438 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.365431 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-workload-socket\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.367081 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.367054 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-istio-envoy\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.367432 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.367407 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4764645c-c41c-4085-8131-424e1ff46c72-istio-podinfo\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.390380 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.390343 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4764645c-c41c-4085-8131-424e1ff46c72-istio-token\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.390540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.390411 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swglv\" (UniqueName: \"kubernetes.io/projected/4764645c-c41c-4085-8131-424e1ff46c72-kube-api-access-swglv\") pod \"e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.530508 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.530397 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:30.865415 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.865382 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj"] Oct 30 13:41:30.866751 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:41:30.866712 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4764645c_c41c_4085_8131_424e1ff46c72.slice/crio-4cd2dad78f60a9a746ce6630236d63b5debc8974a82c1060da6617193d7f6b5c WatchSource:0}: Error finding container 4cd2dad78f60a9a746ce6630236d63b5debc8974a82c1060da6617193d7f6b5c: Status 404 returned error can't find the container with id 4cd2dad78f60a9a746ce6630236d63b5debc8974a82c1060da6617193d7f6b5c Oct 30 13:41:30.869623 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.869589 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:41:30.869711 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.869657 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:41:30.869711 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:30.869685 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:41:31.850736 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:31.850695 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" event={"ID":"4764645c-c41c-4085-8131-424e1ff46c72","Type":"ContainerStarted","Data":"fc39984eacd1e3c6be52ad2758d8d08e6fe4d4f6ccd953c1a446fe108de4e54a"} Oct 30 13:41:31.851201 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:31.850762 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" event={"ID":"4764645c-c41c-4085-8131-424e1ff46c72","Type":"ContainerStarted","Data":"4cd2dad78f60a9a746ce6630236d63b5debc8974a82c1060da6617193d7f6b5c"} Oct 30 13:41:31.927637 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:31.927580 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" podStartSLOduration=1.927562815 podStartE2EDuration="1.927562815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:31.91296433 +0000 UTC m=+2948.545151485" watchObservedRunningTime="2026-10-30 13:41:31.927562815 +0000 UTC m=+2948.559750058" Oct 30 13:41:32.531450 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:32.531413 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:32.533038 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:32.533010 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 13:41:32.533174 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:32.533067 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" podUID="4764645c-c41c-4085-8131-424e1ff46c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 13:41:33.530845 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:33.530807 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 13:41:33.531291 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:33.530869 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" podUID="4764645c-c41c-4085-8131-424e1ff46c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 13:41:34.531157 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:34.531112 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 13:41:34.531718 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:34.531167 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" podUID="4764645c-c41c-4085-8131-424e1ff46c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 13:41:35.531285 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:35.531251 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 13:41:35.531725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:35.531318 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" podUID="4764645c-c41c-4085-8131-424e1ff46c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 13:41:36.531698 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:36.531655 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 13:41:36.532105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:36.531736 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" podUID="4764645c-c41c-4085-8131-424e1ff46c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 13:41:37.531359 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:37.531316 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 13:41:37.531564 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:37.531398 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" podUID="4764645c-c41c-4085-8131-424e1ff46c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 13:41:37.873287 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:37.873206 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c_ca5ec0d2-744a-4cff-a863-4aae9666eccc/payload-processing/0.log" Oct 30 13:41:38.594879 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:38.594844 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7_57a206b4-1557-4b43-9b49-f49f981334e9/payload-processing/0.log" Oct 30 13:41:38.799230 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:38.799200 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7_57a206b4-1557-4b43-9b49-f49f981334e9/payload-processing/0.log" Oct 30 13:41:39.079084 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:39.079052 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:39.079575 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:39.079532 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:39.080797 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:39.080771 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:41:42.794321 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:42.794283 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794"] Oct 30 13:41:42.800156 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:42.800137 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:42.863545 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:42.856338 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794"] Oct 30 13:41:42.923738 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:42.923703 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:42.923945 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:42.923782 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:42.923945 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:42.923872 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:42.924047 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:42.923975 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:42.924047 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:42.924017 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rtzd\" (UniqueName: \"kubernetes.io/projected/1c94ab46-fc9f-4f4d-899c-423d65bde06e-kube-api-access-7rtzd\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:42.924133 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:42.924106 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:42.924202 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:42.924190 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:42.924240 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:42.924218 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:42.924297 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:42.924280 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.025225 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.025188 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.025430 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.025246 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.025430 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.025309 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.025430 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.025334 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.025430 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.025395 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.025710 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.025433 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rtzd\" (UniqueName: \"kubernetes.io/projected/1c94ab46-fc9f-4f4d-899c-423d65bde06e-kube-api-access-7rtzd\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.025710 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.025556 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.025710 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.025670 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.025846 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.025721 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.025846 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.025730 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-workload-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.026146 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.026117 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-data\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.026263 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.026150 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istiod-ca-cert\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.026263 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.026185 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-credential-socket\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.026263 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.026209 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-workload-certs\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.027755 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.027729 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-envoy\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.028275 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.028255 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-podinfo\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.053657 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.053582 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-token\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.053784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.053755 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rtzd\" (UniqueName: \"kubernetes.io/projected/1c94ab46-fc9f-4f4d-899c-423d65bde06e-kube-api-access-7rtzd\") pod \"e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.115131 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.115099 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:43.467725 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.467698 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794"] Oct 30 13:41:43.468305 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:41:43.468279 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c94ab46_fc9f_4f4d_899c_423d65bde06e.slice/crio-324b23e841b727c948c86abd9a56023f515f7cba851b640dad1070a68a2747d7 WatchSource:0}: Error finding container 324b23e841b727c948c86abd9a56023f515f7cba851b640dad1070a68a2747d7: Status 404 returned error can't find the container with id 324b23e841b727c948c86abd9a56023f515f7cba851b640dad1070a68a2747d7 Oct 30 13:41:43.471213 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.471175 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:41:43.471326 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.471290 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:41:43.471397 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:43.471339 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:41:44.067886 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:44.067842 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" event={"ID":"1c94ab46-fc9f-4f4d-899c-423d65bde06e","Type":"ContainerStarted","Data":"87fe446182951d6c9801ff291312fa5827447e51e35324e8230fb77decc6fbb1"} Oct 30 13:41:44.068317 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:44.067901 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" event={"ID":"1c94ab46-fc9f-4f4d-899c-423d65bde06e","Type":"ContainerStarted","Data":"324b23e841b727c948c86abd9a56023f515f7cba851b640dad1070a68a2747d7"} Oct 30 13:41:44.116012 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:44.115971 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:44.117821 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:44.117793 7010 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 13:41:44.117951 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:44.117840 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" podUID="1c94ab46-fc9f-4f4d-899c-423d65bde06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 13:41:44.129074 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:44.129022 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" podStartSLOduration=2.129007671 podStartE2EDuration="2.129007671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:44.124675233 +0000 UTC m=+2960.756862395" watchObservedRunningTime="2026-10-30 13:41:44.129007671 +0000 UTC m=+2960.761194832" Oct 30 13:41:45.116607 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:45.116570 7010 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 13:41:45.117010 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:45.116632 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" podUID="1c94ab46-fc9f-4f4d-899c-423d65bde06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 13:41:46.115748 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:46.115715 7010 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 13:41:46.115939 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:46.115774 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" podUID="1c94ab46-fc9f-4f4d-899c-423d65bde06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 13:41:47.116555 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:47.116480 7010 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 13:41:47.116962 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:47.116581 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" podUID="1c94ab46-fc9f-4f4d-899c-423d65bde06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 13:41:48.115462 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:48.115406 7010 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 13:41:48.115697 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:48.115479 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" podUID="1c94ab46-fc9f-4f4d-899c-423d65bde06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 13:41:49.116507 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:49.116454 7010 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 13:41:49.116917 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:49.116538 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" podUID="1c94ab46-fc9f-4f4d-899c-423d65bde06e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 13:41:50.121519 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:50.121474 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:50.122030 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:50.122009 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:50.122752 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:50.122729 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:41:52.651013 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:52.650979 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn"] Oct 30 13:41:52.651534 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:52.651186 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" podUID="37d669cd-c104-458a-85c2-65d8f7e0c50a" containerName="istio-proxy" containerID="cri-o://03d17a3ee8584b75c10c98806a8756a141d70b43e174a4df452d6df2f5102ab8" gracePeriod=30 Oct 30 13:41:57.938084 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:57.938057 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:41:58.112597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.112553 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-workload-socket\") pod \"37d669cd-c104-458a-85c2-65d8f7e0c50a\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " Oct 30 13:41:58.112769 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.112632 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-token\" (UniqueName: \"kubernetes.io/projected/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-token\") pod \"37d669cd-c104-458a-85c2-65d8f7e0c50a\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " Oct 30 13:41:58.112820 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.112762 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d669cd-c104-458a-85c2-65d8f7e0c50a-kube-api-access-khdxv\" (UniqueName: \"kubernetes.io/projected/37d669cd-c104-458a-85c2-65d8f7e0c50a-kube-api-access-khdxv\") pod \"37d669cd-c104-458a-85c2-65d8f7e0c50a\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " Oct 30 13:41:58.112883 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.112818 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-workload-socket" pod "37d669cd-c104-458a-85c2-65d8f7e0c50a" (UID: "37d669cd-c104-458a-85c2-65d8f7e0c50a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:58.112883 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.112854 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-workload-certs\") pod \"37d669cd-c104-458a-85c2-65d8f7e0c50a\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " Oct 30 13:41:58.113002 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.112884 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-data\") pod \"37d669cd-c104-458a-85c2-65d8f7e0c50a\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " Oct 30 13:41:58.113002 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.112978 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d669cd-c104-458a-85c2-65d8f7e0c50a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d669cd-c104-458a-85c2-65d8f7e0c50a-istiod-ca-cert\") pod \"37d669cd-c104-458a-85c2-65d8f7e0c50a\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " Oct 30 13:41:58.113112 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.113050 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-workload-certs" pod "37d669cd-c104-458a-85c2-65d8f7e0c50a" (UID: "37d669cd-c104-458a-85c2-65d8f7e0c50a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:58.113112 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.113075 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-podinfo\") pod \"37d669cd-c104-458a-85c2-65d8f7e0c50a\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " Oct 30 13:41:58.113213 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.113125 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-envoy\") pod \"37d669cd-c104-458a-85c2-65d8f7e0c50a\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " Oct 30 13:41:58.113213 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.113162 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-credential-socket\") pod \"37d669cd-c104-458a-85c2-65d8f7e0c50a\" (UID: \"37d669cd-c104-458a-85c2-65d8f7e0c50a\") " Oct 30 13:41:58.113213 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.113173 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-data" pod "37d669cd-c104-458a-85c2-65d8f7e0c50a" (UID: "37d669cd-c104-458a-85c2-65d8f7e0c50a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:58.113394 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.113363 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d669cd-c104-458a-85c2-65d8f7e0c50a-istiod-ca-cert" pod "37d669cd-c104-458a-85c2-65d8f7e0c50a" (UID: "37d669cd-c104-458a-85c2-65d8f7e0c50a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:41:58.113521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.113484 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-credential-socket" pod "37d669cd-c104-458a-85c2-65d8f7e0c50a" (UID: "37d669cd-c104-458a-85c2-65d8f7e0c50a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:58.113825 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.113800 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:58.113825 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.113822 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:58.113936 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.113836 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:58.113936 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.113849 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d669cd-c104-458a-85c2-65d8f7e0c50a-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:58.113936 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.113861 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:58.115108 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.115079 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d669cd-c104-458a-85c2-65d8f7e0c50a-kube-api-access-khdxv" pod "37d669cd-c104-458a-85c2-65d8f7e0c50a" (UID: "37d669cd-c104-458a-85c2-65d8f7e0c50a"). InnerVolumeSpecName "kube-api-access-khdxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:58.115322 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.115304 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-podinfo" pod "37d669cd-c104-458a-85c2-65d8f7e0c50a" (UID: "37d669cd-c104-458a-85c2-65d8f7e0c50a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:41:58.115445 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.115414 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-envoy" pod "37d669cd-c104-458a-85c2-65d8f7e0c50a" (UID: "37d669cd-c104-458a-85c2-65d8f7e0c50a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:41:58.115513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.115476 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-token" pod "37d669cd-c104-458a-85c2-65d8f7e0c50a" (UID: "37d669cd-c104-458a-85c2-65d8f7e0c50a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:41:58.221159 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.221055 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:58.221443 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.221414 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khdxv\" (UniqueName: \"kubernetes.io/projected/37d669cd-c104-458a-85c2-65d8f7e0c50a-kube-api-access-khdxv\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:58.221632 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.221615 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:58.221731 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.221720 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d669cd-c104-458a-85c2-65d8f7e0c50a-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:41:58.237673 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.237593 7010 generic.go:356] "Generic (PLEG): container finished" podID="37d669cd-c104-458a-85c2-65d8f7e0c50a" containerID="03d17a3ee8584b75c10c98806a8756a141d70b43e174a4df452d6df2f5102ab8" exitCode=0 Oct 30 13:41:58.238110 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.238082 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" event={"ID":"37d669cd-c104-458a-85c2-65d8f7e0c50a","Type":"ContainerDied","Data":"03d17a3ee8584b75c10c98806a8756a141d70b43e174a4df452d6df2f5102ab8"} Oct 30 13:41:58.238303 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.238285 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" event={"ID":"37d669cd-c104-458a-85c2-65d8f7e0c50a","Type":"ContainerDied","Data":"9d7e2bd1c39f707b7d52187ce2af621d48aafb57e11f3441dd6b0c47d008e4aa"} Oct 30 13:41:58.238436 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.238423 7010 scope.go:122] "RemoveContainer" containerID="03d17a3ee8584b75c10c98806a8756a141d70b43e174a4df452d6df2f5102ab8" Oct 30 13:41:58.241652 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.241576 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn" Oct 30 13:41:58.258216 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.258190 7010 scope.go:122] "RemoveContainer" containerID="03d17a3ee8584b75c10c98806a8756a141d70b43e174a4df452d6df2f5102ab8" Oct 30 13:41:58.258579 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:41:58.258561 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d17a3ee8584b75c10c98806a8756a141d70b43e174a4df452d6df2f5102ab8\": container with ID starting with 03d17a3ee8584b75c10c98806a8756a141d70b43e174a4df452d6df2f5102ab8 not found: ID does not exist" containerID="03d17a3ee8584b75c10c98806a8756a141d70b43e174a4df452d6df2f5102ab8" Oct 30 13:41:58.258630 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.258590 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d17a3ee8584b75c10c98806a8756a141d70b43e174a4df452d6df2f5102ab8"} err="failed to get container status \"03d17a3ee8584b75c10c98806a8756a141d70b43e174a4df452d6df2f5102ab8\": rpc error: code = NotFound desc = could not find container \"03d17a3ee8584b75c10c98806a8756a141d70b43e174a4df452d6df2f5102ab8\": container with ID starting with 03d17a3ee8584b75c10c98806a8756a141d70b43e174a4df452d6df2f5102ab8 not found: ID does not exist" Oct 30 13:41:58.298332 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.298296 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn"] Oct 30 13:41:58.302323 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:58.302292 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-zfjhn"] Oct 30 13:41:59.402761 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:41:59.402718 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:41:59.491653 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:41:59.491618 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d669cd-c104-458a-85c2-65d8f7e0c50a" path="/var/lib/kubelet/pods/37d669cd-c104-458a-85c2-65d8f7e0c50a/volumes" Oct 30 13:42:04.814234 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:04.814148 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg"] Oct 30 13:42:04.814743 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:04.814353 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" podUID="5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" containerName="istio-proxy" containerID="cri-o://c0c43b907461ba520f28a6d93ab22d68da228512bac0dc97f5d2060e0cbcb5c4" gracePeriod=30 Oct 30 13:42:09.898460 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:42:09.898421 7010 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f15b5e8_0877_4f50_ae90_6bfaab9a5fb5.slice/crio-c0c43b907461ba520f28a6d93ab22d68da228512bac0dc97f5d2060e0cbcb5c4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f15b5e8_0877_4f50_ae90_6bfaab9a5fb5.slice/crio-conmon-c0c43b907461ba520f28a6d93ab22d68da228512bac0dc97f5d2060e0cbcb5c4.scope\": RecentStats: unable to find data in memory cache]" Oct 30 13:42:10.102435 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.102403 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:42:10.219894 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.219853 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-podinfo\") pod \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " Oct 30 13:42:10.220114 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.219979 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-credential-socket\") pod \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " Oct 30 13:42:10.220114 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.220024 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-envoy\") pod \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " Oct 30 13:42:10.220114 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.220053 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-workload-socket\") pod \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " Oct 30 13:42:10.220114 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.220094 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-data\") pod \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " Oct 30 13:42:10.220351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.220127 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-workload-certs\") pod \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " Oct 30 13:42:10.220351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.220157 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istiod-ca-cert\") pod \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " Oct 30 13:42:10.220351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.220215 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-token\" (UniqueName: \"kubernetes.io/projected/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-token\") pod \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " Oct 30 13:42:10.220351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.220343 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-kube-api-access-7pkch\" (UniqueName: \"kubernetes.io/projected/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-kube-api-access-7pkch\") pod \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\" (UID: \"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5\") " Oct 30 13:42:10.220651 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.220356 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-credential-socket" pod "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" (UID: "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:10.220651 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.220429 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-data" pod "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" (UID: "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:10.220651 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.220446 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-workload-socket" pod "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" (UID: "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:10.220651 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.220585 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istiod-ca-cert" pod "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" (UID: "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:10.220862 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.220791 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-workload-certs" pod "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" (UID: "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:10.221144 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.221117 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:10.221325 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.221300 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:10.221435 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.221329 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:10.221435 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.221344 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:10.221435 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.221357 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:10.222464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.222439 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-envoy" pod "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" (UID: "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:10.222568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.222462 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-podinfo" pod "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" (UID: "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:42:10.222688 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.222666 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-kube-api-access-7pkch" pod "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" (UID: "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5"). InnerVolumeSpecName "kube-api-access-7pkch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:10.222742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.222704 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-token" pod "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" (UID: "5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:10.322873 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.322837 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:10.322873 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.322874 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:10.323076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.322891 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:10.323076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.322905 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pkch\" (UniqueName: \"kubernetes.io/projected/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5-kube-api-access-7pkch\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:10.337602 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.337508 7010 generic.go:356] "Generic (PLEG): container finished" podID="5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" containerID="c0c43b907461ba520f28a6d93ab22d68da228512bac0dc97f5d2060e0cbcb5c4" exitCode=0 Oct 30 13:42:10.337602 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.337551 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" event={"ID":"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5","Type":"ContainerDied","Data":"c0c43b907461ba520f28a6d93ab22d68da228512bac0dc97f5d2060e0cbcb5c4"} Oct 30 13:42:10.337602 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.337591 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" Oct 30 13:42:10.337881 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.337642 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg" event={"ID":"5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5","Type":"ContainerDied","Data":"86e34be7bd0204f2576fef3c94727a515a23f42da949c2a685d518831cdf248d"} Oct 30 13:42:10.337881 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.337655 7010 scope.go:122] "RemoveContainer" containerID="c0c43b907461ba520f28a6d93ab22d68da228512bac0dc97f5d2060e0cbcb5c4" Oct 30 13:42:10.349846 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.349818 7010 scope.go:122] "RemoveContainer" containerID="c0c43b907461ba520f28a6d93ab22d68da228512bac0dc97f5d2060e0cbcb5c4" Oct 30 13:42:10.350211 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:42:10.350188 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c43b907461ba520f28a6d93ab22d68da228512bac0dc97f5d2060e0cbcb5c4\": container with ID starting with c0c43b907461ba520f28a6d93ab22d68da228512bac0dc97f5d2060e0cbcb5c4 not found: ID does not exist" containerID="c0c43b907461ba520f28a6d93ab22d68da228512bac0dc97f5d2060e0cbcb5c4" Oct 30 13:42:10.350289 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.350223 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c43b907461ba520f28a6d93ab22d68da228512bac0dc97f5d2060e0cbcb5c4"} err="failed to get container status \"c0c43b907461ba520f28a6d93ab22d68da228512bac0dc97f5d2060e0cbcb5c4\": rpc error: code = NotFound desc = could not find container \"c0c43b907461ba520f28a6d93ab22d68da228512bac0dc97f5d2060e0cbcb5c4\": container with ID starting with c0c43b907461ba520f28a6d93ab22d68da228512bac0dc97f5d2060e0cbcb5c4 not found: ID does not exist" Oct 30 13:42:10.394283 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.394247 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg"] Oct 30 13:42:10.398612 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:10.398579 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ld7hg"] Oct 30 13:42:11.489687 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:11.489653 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" path="/var/lib/kubelet/pods/5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5/volumes" Oct 30 13:42:21.517461 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:21.515876 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq"] Oct 30 13:42:21.517461 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:21.516238 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" podUID="aa83aaa6-483b-4ae1-aaaa-2f30faf7f576" containerName="payload-processing" containerID="cri-o://6397edabf904d17b04777c80a67a09b9e856e37e41170395028acca2d266d9df" gracePeriod=30 Oct 30 13:42:21.535465 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:21.535434 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2"] Oct 30 13:42:21.535697 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:21.535672 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" podUID="84d2121e-3ecd-4540-889b-3afd02a60f91" containerName="payload-pre-processing" containerID="cri-o://56a0a01fd748b14d5167aba1fb4124dbde558e65e580633f41b584692fcffa09" gracePeriod=30 Oct 30 13:42:22.107831 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.107800 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" Oct 30 13:42:22.111592 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.111566 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" Oct 30 13:42:22.162895 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.162849 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa83aaa6-483b-4ae1-aaaa-2f30faf7f576-kube-api-access-rpfst\" (UniqueName: \"kubernetes.io/projected/aa83aaa6-483b-4ae1-aaaa-2f30faf7f576-kube-api-access-rpfst\") pod \"aa83aaa6-483b-4ae1-aaaa-2f30faf7f576\" (UID: \"aa83aaa6-483b-4ae1-aaaa-2f30faf7f576\") " Oct 30 13:42:22.163105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.162908 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84d2121e-3ecd-4540-889b-3afd02a60f91-kube-api-access-gmd87\" (UniqueName: \"kubernetes.io/projected/84d2121e-3ecd-4540-889b-3afd02a60f91-kube-api-access-gmd87\") pod \"84d2121e-3ecd-4540-889b-3afd02a60f91\" (UID: \"84d2121e-3ecd-4540-889b-3afd02a60f91\") " Oct 30 13:42:22.163105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.163063 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa83aaa6-483b-4ae1-aaaa-2f30faf7f576-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa83aaa6-483b-4ae1-aaaa-2f30faf7f576-plugins-config-volume\") pod \"aa83aaa6-483b-4ae1-aaaa-2f30faf7f576\" (UID: \"aa83aaa6-483b-4ae1-aaaa-2f30faf7f576\") " Oct 30 13:42:22.163241 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.163215 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84d2121e-3ecd-4540-889b-3afd02a60f91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d2121e-3ecd-4540-889b-3afd02a60f91-plugins-config-volume\") pod \"84d2121e-3ecd-4540-889b-3afd02a60f91\" (UID: \"84d2121e-3ecd-4540-889b-3afd02a60f91\") " Oct 30 13:42:22.164057 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.164023 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d2121e-3ecd-4540-889b-3afd02a60f91-plugins-config-volume" pod "84d2121e-3ecd-4540-889b-3afd02a60f91" (UID: "84d2121e-3ecd-4540-889b-3afd02a60f91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:22.164722 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.164672 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa83aaa6-483b-4ae1-aaaa-2f30faf7f576-plugins-config-volume" pod "aa83aaa6-483b-4ae1-aaaa-2f30faf7f576" (UID: "aa83aaa6-483b-4ae1-aaaa-2f30faf7f576"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:22.166387 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.166360 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa83aaa6-483b-4ae1-aaaa-2f30faf7f576-kube-api-access-rpfst" pod "aa83aaa6-483b-4ae1-aaaa-2f30faf7f576" (UID: "aa83aaa6-483b-4ae1-aaaa-2f30faf7f576"). InnerVolumeSpecName "kube-api-access-rpfst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:22.166624 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.166601 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d2121e-3ecd-4540-889b-3afd02a60f91-kube-api-access-gmd87" pod "84d2121e-3ecd-4540-889b-3afd02a60f91" (UID: "84d2121e-3ecd-4540-889b-3afd02a60f91"). InnerVolumeSpecName "kube-api-access-gmd87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:22.265792 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.265753 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d2121e-3ecd-4540-889b-3afd02a60f91-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:22.265792 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.265790 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpfst\" (UniqueName: \"kubernetes.io/projected/aa83aaa6-483b-4ae1-aaaa-2f30faf7f576-kube-api-access-rpfst\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:22.266182 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.265804 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmd87\" (UniqueName: \"kubernetes.io/projected/84d2121e-3ecd-4540-889b-3afd02a60f91-kube-api-access-gmd87\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:22.266182 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.265817 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa83aaa6-483b-4ae1-aaaa-2f30faf7f576-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:22.485616 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.485581 7010 generic.go:356] "Generic (PLEG): container finished" podID="84d2121e-3ecd-4540-889b-3afd02a60f91" containerID="56a0a01fd748b14d5167aba1fb4124dbde558e65e580633f41b584692fcffa09" exitCode=0 Oct 30 13:42:22.485839 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.485629 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" Oct 30 13:42:22.485839 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.485674 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" event={"ID":"84d2121e-3ecd-4540-889b-3afd02a60f91","Type":"ContainerDied","Data":"56a0a01fd748b14d5167aba1fb4124dbde558e65e580633f41b584692fcffa09"} Oct 30 13:42:22.485839 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.485824 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2" event={"ID":"84d2121e-3ecd-4540-889b-3afd02a60f91","Type":"ContainerDied","Data":"752e6a402ba5677ce396c47f457a1bc0f2bf76c15dae2edefd4b89f50ab3f744"} Oct 30 13:42:22.486023 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.485858 7010 scope.go:122] "RemoveContainer" containerID="56a0a01fd748b14d5167aba1fb4124dbde558e65e580633f41b584692fcffa09" Oct 30 13:42:22.487245 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.487217 7010 generic.go:356] "Generic (PLEG): container finished" podID="aa83aaa6-483b-4ae1-aaaa-2f30faf7f576" containerID="6397edabf904d17b04777c80a67a09b9e856e37e41170395028acca2d266d9df" exitCode=0 Oct 30 13:42:22.487245 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.487251 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" Oct 30 13:42:22.487424 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.487285 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" event={"ID":"aa83aaa6-483b-4ae1-aaaa-2f30faf7f576","Type":"ContainerDied","Data":"6397edabf904d17b04777c80a67a09b9e856e37e41170395028acca2d266d9df"} Oct 30 13:42:22.487424 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.487347 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq" event={"ID":"aa83aaa6-483b-4ae1-aaaa-2f30faf7f576","Type":"ContainerDied","Data":"aef5f1afd9bee875350276f34ed7bb62d248535c4ba9f4157131104d4880307a"} Oct 30 13:42:22.504876 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.504852 7010 scope.go:122] "RemoveContainer" containerID="56a0a01fd748b14d5167aba1fb4124dbde558e65e580633f41b584692fcffa09" Oct 30 13:42:22.505246 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:42:22.505225 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56a0a01fd748b14d5167aba1fb4124dbde558e65e580633f41b584692fcffa09\": container with ID starting with 56a0a01fd748b14d5167aba1fb4124dbde558e65e580633f41b584692fcffa09 not found: ID does not exist" containerID="56a0a01fd748b14d5167aba1fb4124dbde558e65e580633f41b584692fcffa09" Oct 30 13:42:22.505365 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.505255 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56a0a01fd748b14d5167aba1fb4124dbde558e65e580633f41b584692fcffa09"} err="failed to get container status \"56a0a01fd748b14d5167aba1fb4124dbde558e65e580633f41b584692fcffa09\": rpc error: code = NotFound desc = could not find container \"56a0a01fd748b14d5167aba1fb4124dbde558e65e580633f41b584692fcffa09\": container with ID starting with 56a0a01fd748b14d5167aba1fb4124dbde558e65e580633f41b584692fcffa09 not found: ID does not exist" Oct 30 13:42:22.505365 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.505275 7010 scope.go:122] "RemoveContainer" containerID="6397edabf904d17b04777c80a67a09b9e856e37e41170395028acca2d266d9df" Oct 30 13:42:22.517938 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.517914 7010 scope.go:122] "RemoveContainer" containerID="6397edabf904d17b04777c80a67a09b9e856e37e41170395028acca2d266d9df" Oct 30 13:42:22.518303 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:42:22.518282 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6397edabf904d17b04777c80a67a09b9e856e37e41170395028acca2d266d9df\": container with ID starting with 6397edabf904d17b04777c80a67a09b9e856e37e41170395028acca2d266d9df not found: ID does not exist" containerID="6397edabf904d17b04777c80a67a09b9e856e37e41170395028acca2d266d9df" Oct 30 13:42:22.518361 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.518313 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6397edabf904d17b04777c80a67a09b9e856e37e41170395028acca2d266d9df"} err="failed to get container status \"6397edabf904d17b04777c80a67a09b9e856e37e41170395028acca2d266d9df\": rpc error: code = NotFound desc = could not find container \"6397edabf904d17b04777c80a67a09b9e856e37e41170395028acca2d266d9df\": container with ID starting with 6397edabf904d17b04777c80a67a09b9e856e37e41170395028acca2d266d9df not found: ID does not exist" Oct 30 13:42:22.559791 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.559749 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2"] Oct 30 13:42:22.565764 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.565727 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c54f0f-888c5dcc5-cght2"] Oct 30 13:42:22.603034 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.602987 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq"] Oct 30 13:42:22.605898 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:22.605866 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c54f0f-5f7fbbff8d-vnmdq"] Oct 30 13:42:23.502855 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:23.502815 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84d2121e-3ecd-4540-889b-3afd02a60f91" path="/var/lib/kubelet/pods/84d2121e-3ecd-4540-889b-3afd02a60f91/volumes" Oct 30 13:42:23.503307 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:23.503293 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa83aaa6-483b-4ae1-aaaa-2f30faf7f576" path="/var/lib/kubelet/pods/aa83aaa6-483b-4ae1-aaaa-2f30faf7f576/volumes" Oct 30 13:42:24.021174 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:24.021144 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:42:24.041997 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:24.041969 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:42:28.264360 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.264318 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7"] Oct 30 13:42:28.264908 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.264589 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" podUID="7460fa64-f0e9-4542-a6cd-31d353166cc1" containerName="maas-api" containerID="cri-o://d01290c689d595f75173c06b7d9f72862637a2b170739cb9336574b93807225b" gracePeriod=30 Oct 30 13:42:28.389114 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.389076 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" podUID="7460fa64-f0e9-4542-a6cd-31d353166cc1" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.34:8443/health\": dial tcp 10.134.0.34:8443: connect: connection refused" Oct 30 13:42:28.595536 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.590000 7010 generic.go:356] "Generic (PLEG): container finished" podID="7460fa64-f0e9-4542-a6cd-31d353166cc1" containerID="d01290c689d595f75173c06b7d9f72862637a2b170739cb9336574b93807225b" exitCode=0 Oct 30 13:42:28.595536 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.590181 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" event={"ID":"7460fa64-f0e9-4542-a6cd-31d353166cc1","Type":"ContainerDied","Data":"d01290c689d595f75173c06b7d9f72862637a2b170739cb9336574b93807225b"} Oct 30 13:42:28.595536 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.590240 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" event={"ID":"7460fa64-f0e9-4542-a6cd-31d353166cc1","Type":"ContainerDied","Data":"205c89f8cfd6015c708b727c342a808434f3aa755e9f744d758698903899f4de"} Oct 30 13:42:28.595536 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.590264 7010 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="205c89f8cfd6015c708b727c342a808434f3aa755e9f744d758698903899f4de" Oct 30 13:42:28.599079 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.599041 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:42:28.697322 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.697281 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7460fa64-f0e9-4542-a6cd-31d353166cc1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7460fa64-f0e9-4542-a6cd-31d353166cc1-metrics-certs\") pod \"7460fa64-f0e9-4542-a6cd-31d353166cc1\" (UID: \"7460fa64-f0e9-4542-a6cd-31d353166cc1\") " Oct 30 13:42:28.697654 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.697629 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7460fa64-f0e9-4542-a6cd-31d353166cc1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7460fa64-f0e9-4542-a6cd-31d353166cc1-openshift-service-ca\") pod \"7460fa64-f0e9-4542-a6cd-31d353166cc1\" (UID: \"7460fa64-f0e9-4542-a6cd-31d353166cc1\") " Oct 30 13:42:28.697877 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.697860 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7460fa64-f0e9-4542-a6cd-31d353166cc1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7460fa64-f0e9-4542-a6cd-31d353166cc1-maas-api-tls\") pod \"7460fa64-f0e9-4542-a6cd-31d353166cc1\" (UID: \"7460fa64-f0e9-4542-a6cd-31d353166cc1\") " Oct 30 13:42:28.697969 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.697951 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7460fa64-f0e9-4542-a6cd-31d353166cc1-kube-api-access-w4z5h\" (UniqueName: \"kubernetes.io/projected/7460fa64-f0e9-4542-a6cd-31d353166cc1-kube-api-access-w4z5h\") pod \"7460fa64-f0e9-4542-a6cd-31d353166cc1\" (UID: \"7460fa64-f0e9-4542-a6cd-31d353166cc1\") " Oct 30 13:42:28.698251 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.698211 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7460fa64-f0e9-4542-a6cd-31d353166cc1-openshift-service-ca" pod "7460fa64-f0e9-4542-a6cd-31d353166cc1" (UID: "7460fa64-f0e9-4542-a6cd-31d353166cc1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:28.698776 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.698749 7010 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7460fa64-f0e9-4542-a6cd-31d353166cc1-openshift-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:28.700779 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.700753 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7460fa64-f0e9-4542-a6cd-31d353166cc1-metrics-certs" pod "7460fa64-f0e9-4542-a6cd-31d353166cc1" (UID: "7460fa64-f0e9-4542-a6cd-31d353166cc1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:42:28.701628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.701602 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7460fa64-f0e9-4542-a6cd-31d353166cc1-kube-api-access-w4z5h" pod "7460fa64-f0e9-4542-a6cd-31d353166cc1" (UID: "7460fa64-f0e9-4542-a6cd-31d353166cc1"). InnerVolumeSpecName "kube-api-access-w4z5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:28.703853 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.703824 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7460fa64-f0e9-4542-a6cd-31d353166cc1-maas-api-tls" pod "7460fa64-f0e9-4542-a6cd-31d353166cc1" (UID: "7460fa64-f0e9-4542-a6cd-31d353166cc1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:42:28.802403 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.802357 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7460fa64-f0e9-4542-a6cd-31d353166cc1-maas-api-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:28.802403 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.802401 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4z5h\" (UniqueName: \"kubernetes.io/projected/7460fa64-f0e9-4542-a6cd-31d353166cc1-kube-api-access-w4z5h\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:28.802403 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:28.802415 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7460fa64-f0e9-4542-a6cd-31d353166cc1-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:29.404249 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:42:29.404209 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:42:29.597120 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:29.597084 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7" Oct 30 13:42:29.704582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:29.704501 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7"] Oct 30 13:42:29.717230 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:29.717191 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cb58b8-c59c75f88-smph7"] Oct 30 13:42:31.235661 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.235623 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55"] Oct 30 13:42:31.236275 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.235869 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" podUID="0f1352f2-5503-4593-96d1-2cbbf798145e" containerName="payload-processing" containerID="cri-o://848e9085c854bb1ff69884ea2a1137420212f0224a1d1db854238b9ef56d097b" gracePeriod=30 Oct 30 13:42:31.255351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.255309 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97"] Oct 30 13:42:31.255622 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.255580 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" podUID="5972caf5-43d7-4006-8c85-00d82f960f84" containerName="payload-pre-processing" containerID="cri-o://f3a26631ff753d3a7a725ebab12d570a8ed2380821fcca5b4813d4250b37930d" gracePeriod=30 Oct 30 13:42:31.492835 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.492755 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7460fa64-f0e9-4542-a6cd-31d353166cc1" path="/var/lib/kubelet/pods/7460fa64-f0e9-4542-a6cd-31d353166cc1/volumes" Oct 30 13:42:31.748565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.745975 7010 generic.go:356] "Generic (PLEG): container finished" podID="0f1352f2-5503-4593-96d1-2cbbf798145e" containerID="848e9085c854bb1ff69884ea2a1137420212f0224a1d1db854238b9ef56d097b" exitCode=0 Oct 30 13:42:31.748565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.746094 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" event={"ID":"0f1352f2-5503-4593-96d1-2cbbf798145e","Type":"ContainerDied","Data":"848e9085c854bb1ff69884ea2a1137420212f0224a1d1db854238b9ef56d097b"} Oct 30 13:42:31.748565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.748287 7010 generic.go:356] "Generic (PLEG): container finished" podID="5972caf5-43d7-4006-8c85-00d82f960f84" containerID="f3a26631ff753d3a7a725ebab12d570a8ed2380821fcca5b4813d4250b37930d" exitCode=0 Oct 30 13:42:31.748565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.748339 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" event={"ID":"5972caf5-43d7-4006-8c85-00d82f960f84","Type":"ContainerDied","Data":"f3a26631ff753d3a7a725ebab12d570a8ed2380821fcca5b4813d4250b37930d"} Oct 30 13:42:31.749570 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.749047 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" Oct 30 13:42:31.777688 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.777660 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" Oct 30 13:42:31.841862 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.841830 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f1352f2-5503-4593-96d1-2cbbf798145e-kube-api-access-fl5gj\" (UniqueName: \"kubernetes.io/projected/0f1352f2-5503-4593-96d1-2cbbf798145e-kube-api-access-fl5gj\") pod \"0f1352f2-5503-4593-96d1-2cbbf798145e\" (UID: \"0f1352f2-5503-4593-96d1-2cbbf798145e\") " Oct 30 13:42:31.842027 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.841875 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5972caf5-43d7-4006-8c85-00d82f960f84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5972caf5-43d7-4006-8c85-00d82f960f84-plugins-config-volume\") pod \"5972caf5-43d7-4006-8c85-00d82f960f84\" (UID: \"5972caf5-43d7-4006-8c85-00d82f960f84\") " Oct 30 13:42:31.842027 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.841921 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f1352f2-5503-4593-96d1-2cbbf798145e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1352f2-5503-4593-96d1-2cbbf798145e-plugins-config-volume\") pod \"0f1352f2-5503-4593-96d1-2cbbf798145e\" (UID: \"0f1352f2-5503-4593-96d1-2cbbf798145e\") " Oct 30 13:42:31.842027 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.842005 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5972caf5-43d7-4006-8c85-00d82f960f84-kube-api-access-kxzw2\" (UniqueName: \"kubernetes.io/projected/5972caf5-43d7-4006-8c85-00d82f960f84-kube-api-access-kxzw2\") pod \"5972caf5-43d7-4006-8c85-00d82f960f84\" (UID: \"5972caf5-43d7-4006-8c85-00d82f960f84\") " Oct 30 13:42:31.842385 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.842358 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5972caf5-43d7-4006-8c85-00d82f960f84-plugins-config-volume" pod "5972caf5-43d7-4006-8c85-00d82f960f84" (UID: "5972caf5-43d7-4006-8c85-00d82f960f84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:31.842461 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.842384 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1352f2-5503-4593-96d1-2cbbf798145e-plugins-config-volume" pod "0f1352f2-5503-4593-96d1-2cbbf798145e" (UID: "0f1352f2-5503-4593-96d1-2cbbf798145e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:31.842680 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.842654 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5972caf5-43d7-4006-8c85-00d82f960f84-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:31.842680 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.842681 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1352f2-5503-4593-96d1-2cbbf798145e-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:31.844274 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.844245 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5972caf5-43d7-4006-8c85-00d82f960f84-kube-api-access-kxzw2" pod "5972caf5-43d7-4006-8c85-00d82f960f84" (UID: "5972caf5-43d7-4006-8c85-00d82f960f84"). InnerVolumeSpecName "kube-api-access-kxzw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:31.844361 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.844291 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1352f2-5503-4593-96d1-2cbbf798145e-kube-api-access-fl5gj" pod "0f1352f2-5503-4593-96d1-2cbbf798145e" (UID: "0f1352f2-5503-4593-96d1-2cbbf798145e"). InnerVolumeSpecName "kube-api-access-fl5gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:31.944335 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.944301 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl5gj\" (UniqueName: \"kubernetes.io/projected/0f1352f2-5503-4593-96d1-2cbbf798145e-kube-api-access-fl5gj\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:31.944335 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:31.944332 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxzw2\" (UniqueName: \"kubernetes.io/projected/5972caf5-43d7-4006-8c85-00d82f960f84-kube-api-access-kxzw2\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:32.756549 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:32.756517 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" Oct 30 13:42:32.757039 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:32.756515 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97" event={"ID":"5972caf5-43d7-4006-8c85-00d82f960f84","Type":"ContainerDied","Data":"460f7a2987fc4d843f27f643217ef6ee8392bc62d05eb56b9831a0d9044bb107"} Oct 30 13:42:32.757039 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:32.756659 7010 scope.go:122] "RemoveContainer" containerID="f3a26631ff753d3a7a725ebab12d570a8ed2380821fcca5b4813d4250b37930d" Oct 30 13:42:32.762768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:32.762736 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" event={"ID":"0f1352f2-5503-4593-96d1-2cbbf798145e","Type":"ContainerDied","Data":"6514b81992372020c5cdcdbeb81e32cb4d6a0c7c4cb13f8a6ceaa7a7f60d86b8"} Oct 30 13:42:32.762959 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:32.762933 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55" Oct 30 13:42:32.769694 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:32.769663 7010 scope.go:122] "RemoveContainer" containerID="848e9085c854bb1ff69884ea2a1137420212f0224a1d1db854238b9ef56d097b" Oct 30 13:42:32.826729 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:32.826681 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97"] Oct 30 13:42:32.839346 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:32.839311 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8d3b8-6cff565979-xgj97"] Oct 30 13:42:32.868581 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:32.868544 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55"] Oct 30 13:42:32.874713 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:32.874685 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8d3b8-6f4c855758-5gc55"] Oct 30 13:42:33.490529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:33.490476 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f1352f2-5503-4593-96d1-2cbbf798145e" path="/var/lib/kubelet/pods/0f1352f2-5503-4593-96d1-2cbbf798145e/volumes" Oct 30 13:42:33.490986 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:33.490971 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5972caf5-43d7-4006-8c85-00d82f960f84" path="/var/lib/kubelet/pods/5972caf5-43d7-4006-8c85-00d82f960f84/volumes" Oct 30 13:42:35.948134 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:35.948076 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj"] Oct 30 13:42:35.948812 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:35.948332 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" podUID="ad23e4ff-651b-4e54-a180-d1cc7c73c51d" containerName="payload-processing" containerID="cri-o://b7c05fdb5a7714af87760713f904aceb91f618bc85fee2af41da19cf82941b8d" gracePeriod=30 Oct 30 13:42:35.982625 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:35.982590 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6"] Oct 30 13:42:35.982881 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:35.982851 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" podUID="e51e863c-9c51-4df1-af43-7877a1e2c227" containerName="payload-pre-processing" containerID="cri-o://2a0c3e53b8df63a356fb2c3204e85a9ef29c7acfb714a3539d9dc1b241dd55d0" gracePeriod=30 Oct 30 13:42:36.326541 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.321602 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" Oct 30 13:42:36.412511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.409003 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad23e4ff-651b-4e54-a180-d1cc7c73c51d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad23e4ff-651b-4e54-a180-d1cc7c73c51d-plugins-config-volume\") pod \"ad23e4ff-651b-4e54-a180-d1cc7c73c51d\" (UID: \"ad23e4ff-651b-4e54-a180-d1cc7c73c51d\") " Oct 30 13:42:36.412511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.409275 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad23e4ff-651b-4e54-a180-d1cc7c73c51d-kube-api-access-dm7qh\" (UniqueName: \"kubernetes.io/projected/ad23e4ff-651b-4e54-a180-d1cc7c73c51d-kube-api-access-dm7qh\") pod \"ad23e4ff-651b-4e54-a180-d1cc7c73c51d\" (UID: \"ad23e4ff-651b-4e54-a180-d1cc7c73c51d\") " Oct 30 13:42:36.427776 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.419074 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad23e4ff-651b-4e54-a180-d1cc7c73c51d-plugins-config-volume" pod "ad23e4ff-651b-4e54-a180-d1cc7c73c51d" (UID: "ad23e4ff-651b-4e54-a180-d1cc7c73c51d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:36.433192 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.428741 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad23e4ff-651b-4e54-a180-d1cc7c73c51d-kube-api-access-dm7qh" pod "ad23e4ff-651b-4e54-a180-d1cc7c73c51d" (UID: "ad23e4ff-651b-4e54-a180-d1cc7c73c51d"). InnerVolumeSpecName "kube-api-access-dm7qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:36.480399 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.480318 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" Oct 30 13:42:36.510859 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.510819 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e51e863c-9c51-4df1-af43-7877a1e2c227-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51e863c-9c51-4df1-af43-7877a1e2c227-plugins-config-volume\") pod \"e51e863c-9c51-4df1-af43-7877a1e2c227\" (UID: \"e51e863c-9c51-4df1-af43-7877a1e2c227\") " Oct 30 13:42:36.511142 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.511123 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51e863c-9c51-4df1-af43-7877a1e2c227-kube-api-access-slrlp\" (UniqueName: \"kubernetes.io/projected/e51e863c-9c51-4df1-af43-7877a1e2c227-kube-api-access-slrlp\") pod \"e51e863c-9c51-4df1-af43-7877a1e2c227\" (UID: \"e51e863c-9c51-4df1-af43-7877a1e2c227\") " Oct 30 13:42:36.511354 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.511320 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51e863c-9c51-4df1-af43-7877a1e2c227-plugins-config-volume" pod "e51e863c-9c51-4df1-af43-7877a1e2c227" (UID: "e51e863c-9c51-4df1-af43-7877a1e2c227"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:36.512053 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.511988 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad23e4ff-651b-4e54-a180-d1cc7c73c51d-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:36.512053 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.512015 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51e863c-9c51-4df1-af43-7877a1e2c227-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:36.512053 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.512029 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm7qh\" (UniqueName: \"kubernetes.io/projected/ad23e4ff-651b-4e54-a180-d1cc7c73c51d-kube-api-access-dm7qh\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:36.513855 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.513830 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51e863c-9c51-4df1-af43-7877a1e2c227-kube-api-access-slrlp" pod "e51e863c-9c51-4df1-af43-7877a1e2c227" (UID: "e51e863c-9c51-4df1-af43-7877a1e2c227"). InnerVolumeSpecName "kube-api-access-slrlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:36.614088 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.614048 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slrlp\" (UniqueName: \"kubernetes.io/projected/e51e863c-9c51-4df1-af43-7877a1e2c227-kube-api-access-slrlp\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:36.668280 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.668246 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj"] Oct 30 13:42:36.668553 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.668520 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" podUID="4764645c-c41c-4085-8131-424e1ff46c72" containerName="istio-proxy" containerID="cri-o://fc39984eacd1e3c6be52ad2758d8d08e6fe4d4f6ccd953c1a446fe108de4e54a" gracePeriod=30 Oct 30 13:42:36.831959 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.823298 7010 generic.go:356] "Generic (PLEG): container finished" podID="e51e863c-9c51-4df1-af43-7877a1e2c227" containerID="2a0c3e53b8df63a356fb2c3204e85a9ef29c7acfb714a3539d9dc1b241dd55d0" exitCode=0 Oct 30 13:42:36.831959 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.823529 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" event={"ID":"e51e863c-9c51-4df1-af43-7877a1e2c227","Type":"ContainerDied","Data":"2a0c3e53b8df63a356fb2c3204e85a9ef29c7acfb714a3539d9dc1b241dd55d0"} Oct 30 13:42:36.831959 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.823585 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" event={"ID":"e51e863c-9c51-4df1-af43-7877a1e2c227","Type":"ContainerDied","Data":"7f2367792fa5e4bd7ce74396fa927f80069181fc5ca9277731fc80be2885216f"} Oct 30 13:42:36.831959 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.823617 7010 scope.go:122] "RemoveContainer" containerID="2a0c3e53b8df63a356fb2c3204e85a9ef29c7acfb714a3539d9dc1b241dd55d0" Oct 30 13:42:36.831959 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.823862 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6" Oct 30 13:42:36.850734 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.850604 7010 generic.go:356] "Generic (PLEG): container finished" podID="ad23e4ff-651b-4e54-a180-d1cc7c73c51d" containerID="b7c05fdb5a7714af87760713f904aceb91f618bc85fee2af41da19cf82941b8d" exitCode=0 Oct 30 13:42:36.850830 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.850763 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" event={"ID":"ad23e4ff-651b-4e54-a180-d1cc7c73c51d","Type":"ContainerDied","Data":"b7c05fdb5a7714af87760713f904aceb91f618bc85fee2af41da19cf82941b8d"} Oct 30 13:42:36.850830 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.850810 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" event={"ID":"ad23e4ff-651b-4e54-a180-d1cc7c73c51d","Type":"ContainerDied","Data":"5c1e66b0f8a65021c9d4b3bd634a890d33a18e092776e06e89d873332c8edd7d"} Oct 30 13:42:36.850940 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.850879 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj" Oct 30 13:42:36.946229 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.946203 7010 scope.go:122] "RemoveContainer" containerID="2a0c3e53b8df63a356fb2c3204e85a9ef29c7acfb714a3539d9dc1b241dd55d0" Oct 30 13:42:36.948476 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:42:36.948440 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0c3e53b8df63a356fb2c3204e85a9ef29c7acfb714a3539d9dc1b241dd55d0\": container with ID starting with 2a0c3e53b8df63a356fb2c3204e85a9ef29c7acfb714a3539d9dc1b241dd55d0 not found: ID does not exist" containerID="2a0c3e53b8df63a356fb2c3204e85a9ef29c7acfb714a3539d9dc1b241dd55d0" Oct 30 13:42:36.948915 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.948507 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0c3e53b8df63a356fb2c3204e85a9ef29c7acfb714a3539d9dc1b241dd55d0"} err="failed to get container status \"2a0c3e53b8df63a356fb2c3204e85a9ef29c7acfb714a3539d9dc1b241dd55d0\": rpc error: code = NotFound desc = could not find container \"2a0c3e53b8df63a356fb2c3204e85a9ef29c7acfb714a3539d9dc1b241dd55d0\": container with ID starting with 2a0c3e53b8df63a356fb2c3204e85a9ef29c7acfb714a3539d9dc1b241dd55d0 not found: ID does not exist" Oct 30 13:42:36.948915 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.948535 7010 scope.go:122] "RemoveContainer" containerID="b7c05fdb5a7714af87760713f904aceb91f618bc85fee2af41da19cf82941b8d" Oct 30 13:42:36.960533 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.960501 7010 scope.go:122] "RemoveContainer" containerID="b7c05fdb5a7714af87760713f904aceb91f618bc85fee2af41da19cf82941b8d" Oct 30 13:42:36.960848 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:42:36.960822 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7c05fdb5a7714af87760713f904aceb91f618bc85fee2af41da19cf82941b8d\": container with ID starting with b7c05fdb5a7714af87760713f904aceb91f618bc85fee2af41da19cf82941b8d not found: ID does not exist" containerID="b7c05fdb5a7714af87760713f904aceb91f618bc85fee2af41da19cf82941b8d" Oct 30 13:42:36.960953 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.960861 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7c05fdb5a7714af87760713f904aceb91f618bc85fee2af41da19cf82941b8d"} err="failed to get container status \"b7c05fdb5a7714af87760713f904aceb91f618bc85fee2af41da19cf82941b8d\": rpc error: code = NotFound desc = could not find container \"b7c05fdb5a7714af87760713f904aceb91f618bc85fee2af41da19cf82941b8d\": container with ID starting with b7c05fdb5a7714af87760713f904aceb91f618bc85fee2af41da19cf82941b8d not found: ID does not exist" Oct 30 13:42:36.964918 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.964822 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj"] Oct 30 13:42:36.967729 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:36.967704 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf3c79-79ccffcb76-7rddj"] Oct 30 13:42:37.004316 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:37.004277 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6"] Oct 30 13:42:37.012270 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:37.012234 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cf3c79-7bc6467655-wqlp6"] Oct 30 13:42:37.490053 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:37.490020 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad23e4ff-651b-4e54-a180-d1cc7c73c51d" path="/var/lib/kubelet/pods/ad23e4ff-651b-4e54-a180-d1cc7c73c51d/volumes" Oct 30 13:42:37.490454 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:37.490440 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e51e863c-9c51-4df1-af43-7877a1e2c227" path="/var/lib/kubelet/pods/e51e863c-9c51-4df1-af43-7877a1e2c227/volumes" Oct 30 13:42:39.083188 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:39.082264 7010 patch_prober.go:29] interesting pod/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:42:39.083188 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:39.082333 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" podUID="4764645c-c41c-4085-8131-424e1ff46c72" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:42:40.305262 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:40.305218 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh"] Oct 30 13:42:40.305780 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:40.305470 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" podUID="a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" containerName="istio-proxy" containerID="cri-o://76e45eb639a8210d98173abdfa875889b824bed2e83eaedc35789ccb760c561a" gracePeriod=30 Oct 30 13:42:41.998466 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:41.998282 7010 generic.go:356] "Generic (PLEG): container finished" podID="4764645c-c41c-4085-8131-424e1ff46c72" containerID="fc39984eacd1e3c6be52ad2758d8d08e6fe4d4f6ccd953c1a446fe108de4e54a" exitCode=0 Oct 30 13:42:41.998466 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:41.998342 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" event={"ID":"4764645c-c41c-4085-8131-424e1ff46c72","Type":"ContainerDied","Data":"fc39984eacd1e3c6be52ad2758d8d08e6fe4d4f6ccd953c1a446fe108de4e54a"} Oct 30 13:42:42.094104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.093934 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:42:42.203941 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.203907 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4764645c-c41c-4085-8131-424e1ff46c72-istio-token\" (UniqueName: \"kubernetes.io/projected/4764645c-c41c-4085-8131-424e1ff46c72-istio-token\") pod \"4764645c-c41c-4085-8131-424e1ff46c72\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " Oct 30 13:42:42.204113 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.203952 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-credential-socket\") pod \"4764645c-c41c-4085-8131-424e1ff46c72\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " Oct 30 13:42:42.204113 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.204060 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-istio-envoy\") pod \"4764645c-c41c-4085-8131-424e1ff46c72\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " Oct 30 13:42:42.204113 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.204094 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-workload-socket\") pod \"4764645c-c41c-4085-8131-424e1ff46c72\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " Oct 30 13:42:42.204280 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.204141 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4764645c-c41c-4085-8131-424e1ff46c72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4764645c-c41c-4085-8131-424e1ff46c72-istio-podinfo\") pod \"4764645c-c41c-4085-8131-424e1ff46c72\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " Oct 30 13:42:42.204280 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.204196 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4764645c-c41c-4085-8131-424e1ff46c72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4764645c-c41c-4085-8131-424e1ff46c72-istiod-ca-cert\") pod \"4764645c-c41c-4085-8131-424e1ff46c72\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " Oct 30 13:42:42.204389 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.204298 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4764645c-c41c-4085-8131-424e1ff46c72-kube-api-access-swglv\" (UniqueName: \"kubernetes.io/projected/4764645c-c41c-4085-8131-424e1ff46c72-kube-api-access-swglv\") pod \"4764645c-c41c-4085-8131-424e1ff46c72\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " Oct 30 13:42:42.204389 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.204364 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-credential-socket" pod "4764645c-c41c-4085-8131-424e1ff46c72" (UID: "4764645c-c41c-4085-8131-424e1ff46c72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:42.204515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.204392 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-istio-data\") pod \"4764645c-c41c-4085-8131-424e1ff46c72\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " Oct 30 13:42:42.204515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.204401 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-workload-socket" pod "4764645c-c41c-4085-8131-424e1ff46c72" (UID: "4764645c-c41c-4085-8131-424e1ff46c72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:42.204636 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.204529 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-workload-certs\") pod \"4764645c-c41c-4085-8131-424e1ff46c72\" (UID: \"4764645c-c41c-4085-8131-424e1ff46c72\") " Oct 30 13:42:42.204761 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.204736 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-istio-data" pod "4764645c-c41c-4085-8131-424e1ff46c72" (UID: "4764645c-c41c-4085-8131-424e1ff46c72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:42.204867 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.204793 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4764645c-c41c-4085-8131-424e1ff46c72-istiod-ca-cert" pod "4764645c-c41c-4085-8131-424e1ff46c72" (UID: "4764645c-c41c-4085-8131-424e1ff46c72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:42.204990 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.204881 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-workload-certs" pod "4764645c-c41c-4085-8131-424e1ff46c72" (UID: "4764645c-c41c-4085-8131-424e1ff46c72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:42.205302 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.205278 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:42.205404 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.205308 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:42.205404 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.205323 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:42.205404 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.205338 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:42.205404 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.205352 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4764645c-c41c-4085-8131-424e1ff46c72-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:42.206421 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.206394 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-istio-envoy" pod "4764645c-c41c-4085-8131-424e1ff46c72" (UID: "4764645c-c41c-4085-8131-424e1ff46c72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:42.206617 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.206586 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4764645c-c41c-4085-8131-424e1ff46c72-istio-token" pod "4764645c-c41c-4085-8131-424e1ff46c72" (UID: "4764645c-c41c-4085-8131-424e1ff46c72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:42.206675 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.206613 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4764645c-c41c-4085-8131-424e1ff46c72-kube-api-access-swglv" pod "4764645c-c41c-4085-8131-424e1ff46c72" (UID: "4764645c-c41c-4085-8131-424e1ff46c72"). InnerVolumeSpecName "kube-api-access-swglv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:42.206675 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.206658 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4764645c-c41c-4085-8131-424e1ff46c72-istio-podinfo" pod "4764645c-c41c-4085-8131-424e1ff46c72" (UID: "4764645c-c41c-4085-8131-424e1ff46c72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:42:42.307338 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.307249 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4764645c-c41c-4085-8131-424e1ff46c72-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:42.307338 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.307287 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4764645c-c41c-4085-8131-424e1ff46c72-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:42.307338 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.307301 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4764645c-c41c-4085-8131-424e1ff46c72-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:42.307338 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:42.307317 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swglv\" (UniqueName: \"kubernetes.io/projected/4764645c-c41c-4085-8131-424e1ff46c72-kube-api-access-swglv\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:43.021506 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:43.018732 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" event={"ID":"4764645c-c41c-4085-8131-424e1ff46c72","Type":"ContainerDied","Data":"4cd2dad78f60a9a746ce6630236d63b5debc8974a82c1060da6617193d7f6b5c"} Oct 30 13:42:43.021506 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:43.018818 7010 scope.go:122] "RemoveContainer" containerID="fc39984eacd1e3c6be52ad2758d8d08e6fe4d4f6ccd953c1a446fe108de4e54a" Oct 30 13:42:43.021506 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:43.018901 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj" Oct 30 13:42:43.086957 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:43.086914 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj"] Oct 30 13:42:43.101216 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:43.101177 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cb58b8-openshift-default-7bd9785c8f-l9vlj"] Oct 30 13:42:43.489391 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:43.489359 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4764645c-c41c-4085-8131-424e1ff46c72" path="/var/lib/kubelet/pods/4764645c-c41c-4085-8131-424e1ff46c72/volumes" Oct 30 13:42:45.581824 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.581759 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:42:45.660098 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.660060 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-workload-certs\") pod \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " Oct 30 13:42:45.660336 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.660116 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-data\") pod \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " Oct 30 13:42:45.660336 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.660159 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-token\" (UniqueName: \"kubernetes.io/projected/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-token\") pod \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " Oct 30 13:42:45.660336 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.660280 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-credential-socket\") pod \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " Oct 30 13:42:45.660336 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.660310 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-envoy\") pod \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " Oct 30 13:42:45.660594 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.660426 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istiod-ca-cert\") pod \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " Oct 30 13:42:45.660594 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.660478 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-kube-api-access-fr6sv\" (UniqueName: \"kubernetes.io/projected/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-kube-api-access-fr6sv\") pod \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " Oct 30 13:42:45.660594 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.660545 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-podinfo\") pod \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " Oct 30 13:42:45.660594 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.660570 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-workload-socket\") pod \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\" (UID: \"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53\") " Oct 30 13:42:45.661641 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.661081 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-data" pod "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" (UID: "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:45.661641 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.661096 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-workload-certs" pod "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" (UID: "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:45.661641 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.661598 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istiod-ca-cert" pod "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" (UID: "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:45.662009 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.661748 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-workload-socket" pod "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" (UID: "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:45.662335 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.662298 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-credential-socket" pod "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" (UID: "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:45.663666 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.663639 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-token" pod "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" (UID: "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:45.663851 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.663827 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-podinfo" pod "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" (UID: "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:42:45.663933 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.663915 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-kube-api-access-fr6sv" pod "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" (UID: "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53"). InnerVolumeSpecName "kube-api-access-fr6sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:45.665069 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.665046 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-envoy" pod "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" (UID: "a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:42:45.762751 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.762716 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr6sv\" (UniqueName: \"kubernetes.io/projected/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-kube-api-access-fr6sv\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:45.762751 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.762747 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:45.762751 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.762757 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:45.763013 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.762766 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:45.763013 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.762777 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:45.763013 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.762785 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:45.763013 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.762792 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:45.763013 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.762810 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:45.763013 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:45.762818 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:46.058212 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:46.058175 7010 generic.go:356] "Generic (PLEG): container finished" podID="a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" containerID="76e45eb639a8210d98173abdfa875889b824bed2e83eaedc35789ccb760c561a" exitCode=0 Oct 30 13:42:46.058404 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:46.058229 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" Oct 30 13:42:46.058404 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:46.058252 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" event={"ID":"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53","Type":"ContainerDied","Data":"76e45eb639a8210d98173abdfa875889b824bed2e83eaedc35789ccb760c561a"} Oct 30 13:42:46.058404 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:46.058301 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh" event={"ID":"a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53","Type":"ContainerDied","Data":"525ac58c56b3ff2e531797b8c6ef0c7db363ff2a9ddabb7646cd27ffa516f6df"} Oct 30 13:42:46.058404 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:46.058324 7010 scope.go:122] "RemoveContainer" containerID="76e45eb639a8210d98173abdfa875889b824bed2e83eaedc35789ccb760c561a" Oct 30 13:42:46.076424 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:46.076399 7010 scope.go:122] "RemoveContainer" containerID="0d30e05091e905667d1a1994a64024b26cd2e7a944a7bb8ee82e87c8be805bc1" Oct 30 13:42:46.086080 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:46.086055 7010 scope.go:122] "RemoveContainer" containerID="76e45eb639a8210d98173abdfa875889b824bed2e83eaedc35789ccb760c561a" Oct 30 13:42:46.086377 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:42:46.086352 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e45eb639a8210d98173abdfa875889b824bed2e83eaedc35789ccb760c561a\": container with ID starting with 76e45eb639a8210d98173abdfa875889b824bed2e83eaedc35789ccb760c561a not found: ID does not exist" containerID="76e45eb639a8210d98173abdfa875889b824bed2e83eaedc35789ccb760c561a" Oct 30 13:42:46.086454 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:46.086385 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e45eb639a8210d98173abdfa875889b824bed2e83eaedc35789ccb760c561a"} err="failed to get container status \"76e45eb639a8210d98173abdfa875889b824bed2e83eaedc35789ccb760c561a\": rpc error: code = NotFound desc = could not find container \"76e45eb639a8210d98173abdfa875889b824bed2e83eaedc35789ccb760c561a\": container with ID starting with 76e45eb639a8210d98173abdfa875889b824bed2e83eaedc35789ccb760c561a not found: ID does not exist" Oct 30 13:42:46.086454 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:46.086408 7010 scope.go:122] "RemoveContainer" containerID="0d30e05091e905667d1a1994a64024b26cd2e7a944a7bb8ee82e87c8be805bc1" Oct 30 13:42:46.086715 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:42:46.086696 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d30e05091e905667d1a1994a64024b26cd2e7a944a7bb8ee82e87c8be805bc1\": container with ID starting with 0d30e05091e905667d1a1994a64024b26cd2e7a944a7bb8ee82e87c8be805bc1 not found: ID does not exist" containerID="0d30e05091e905667d1a1994a64024b26cd2e7a944a7bb8ee82e87c8be805bc1" Oct 30 13:42:46.086754 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:46.086721 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d30e05091e905667d1a1994a64024b26cd2e7a944a7bb8ee82e87c8be805bc1"} err="failed to get container status \"0d30e05091e905667d1a1994a64024b26cd2e7a944a7bb8ee82e87c8be805bc1\": rpc error: code = NotFound desc = could not find container \"0d30e05091e905667d1a1994a64024b26cd2e7a944a7bb8ee82e87c8be805bc1\": container with ID starting with 0d30e05091e905667d1a1994a64024b26cd2e7a944a7bb8ee82e87c8be805bc1 not found: ID does not exist" Oct 30 13:42:46.115547 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:46.115457 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh"] Oct 30 13:42:46.121415 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:46.121389 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a8d3b8-openshift-default-7496bfbf6c-vhrqh"] Oct 30 13:42:47.504549 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:47.497691 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" path="/var/lib/kubelet/pods/a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53/volumes" Oct 30 13:42:47.504549 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:47.499152 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx"] Oct 30 13:42:47.504549 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:47.499375 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" podUID="36b712a0-c94f-4e1d-b769-56b308d4e7b1" containerName="payload-processing" containerID="cri-o://03d057222714b25be11d47ac49b81bebec3df4a9f02406934804df533d35fa89" gracePeriod=30 Oct 30 13:42:47.553752 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:47.549314 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6"] Oct 30 13:42:47.553752 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:47.549582 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" podUID="28dc2205-c86e-4bd3-b364-15b5a3d6aee4" containerName="payload-pre-processing" containerID="cri-o://fe6369a7f76008bf176028b11c1dc380090103006eb14882f40a13b52949794e" gracePeriod=30 Oct 30 13:42:48.028784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.028697 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" Oct 30 13:42:48.077817 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.077789 7010 generic.go:356] "Generic (PLEG): container finished" podID="36b712a0-c94f-4e1d-b769-56b308d4e7b1" containerID="03d057222714b25be11d47ac49b81bebec3df4a9f02406934804df533d35fa89" exitCode=0 Oct 30 13:42:48.077954 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.077922 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" event={"ID":"36b712a0-c94f-4e1d-b769-56b308d4e7b1","Type":"ContainerDied","Data":"03d057222714b25be11d47ac49b81bebec3df4a9f02406934804df533d35fa89"} Oct 30 13:42:48.078008 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.077985 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" event={"ID":"36b712a0-c94f-4e1d-b769-56b308d4e7b1","Type":"ContainerDied","Data":"68dcfbfb748086e70ca853dc384733c78283608d3af932acd33ef4c81170f42c"} Oct 30 13:42:48.078046 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.078008 7010 scope.go:122] "RemoveContainer" containerID="03d057222714b25be11d47ac49b81bebec3df4a9f02406934804df533d35fa89" Oct 30 13:42:48.078093 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.078083 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx" Oct 30 13:42:48.080428 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.080400 7010 generic.go:356] "Generic (PLEG): container finished" podID="28dc2205-c86e-4bd3-b364-15b5a3d6aee4" containerID="fe6369a7f76008bf176028b11c1dc380090103006eb14882f40a13b52949794e" exitCode=0 Oct 30 13:42:48.080845 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.080511 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" event={"ID":"28dc2205-c86e-4bd3-b364-15b5a3d6aee4","Type":"ContainerDied","Data":"fe6369a7f76008bf176028b11c1dc380090103006eb14882f40a13b52949794e"} Oct 30 13:42:48.085570 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.085546 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" Oct 30 13:42:48.091779 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.091760 7010 scope.go:122] "RemoveContainer" containerID="03d057222714b25be11d47ac49b81bebec3df4a9f02406934804df533d35fa89" Oct 30 13:42:48.092223 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:42:48.092203 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d057222714b25be11d47ac49b81bebec3df4a9f02406934804df533d35fa89\": container with ID starting with 03d057222714b25be11d47ac49b81bebec3df4a9f02406934804df533d35fa89 not found: ID does not exist" containerID="03d057222714b25be11d47ac49b81bebec3df4a9f02406934804df533d35fa89" Oct 30 13:42:48.092310 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.092244 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d057222714b25be11d47ac49b81bebec3df4a9f02406934804df533d35fa89"} err="failed to get container status \"03d057222714b25be11d47ac49b81bebec3df4a9f02406934804df533d35fa89\": rpc error: code = NotFound desc = could not find container \"03d057222714b25be11d47ac49b81bebec3df4a9f02406934804df533d35fa89\": container with ID starting with 03d057222714b25be11d47ac49b81bebec3df4a9f02406934804df533d35fa89 not found: ID does not exist" Oct 30 13:42:48.107053 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.107016 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b712a0-c94f-4e1d-b769-56b308d4e7b1-kube-api-access-pvbdh\" (UniqueName: \"kubernetes.io/projected/36b712a0-c94f-4e1d-b769-56b308d4e7b1-kube-api-access-pvbdh\") pod \"36b712a0-c94f-4e1d-b769-56b308d4e7b1\" (UID: \"36b712a0-c94f-4e1d-b769-56b308d4e7b1\") " Oct 30 13:42:48.107316 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.107266 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b712a0-c94f-4e1d-b769-56b308d4e7b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b712a0-c94f-4e1d-b769-56b308d4e7b1-plugins-config-volume\") pod \"36b712a0-c94f-4e1d-b769-56b308d4e7b1\" (UID: \"36b712a0-c94f-4e1d-b769-56b308d4e7b1\") " Oct 30 13:42:48.108792 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.108763 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b712a0-c94f-4e1d-b769-56b308d4e7b1-plugins-config-volume" pod "36b712a0-c94f-4e1d-b769-56b308d4e7b1" (UID: "36b712a0-c94f-4e1d-b769-56b308d4e7b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:48.112274 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.112102 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b712a0-c94f-4e1d-b769-56b308d4e7b1-kube-api-access-pvbdh" pod "36b712a0-c94f-4e1d-b769-56b308d4e7b1" (UID: "36b712a0-c94f-4e1d-b769-56b308d4e7b1"). InnerVolumeSpecName "kube-api-access-pvbdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:48.209024 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.208933 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28dc2205-c86e-4bd3-b364-15b5a3d6aee4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dc2205-c86e-4bd3-b364-15b5a3d6aee4-plugins-config-volume\") pod \"28dc2205-c86e-4bd3-b364-15b5a3d6aee4\" (UID: \"28dc2205-c86e-4bd3-b364-15b5a3d6aee4\") " Oct 30 13:42:48.209024 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.208999 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28dc2205-c86e-4bd3-b364-15b5a3d6aee4-kube-api-access-hvfgx\" (UniqueName: \"kubernetes.io/projected/28dc2205-c86e-4bd3-b364-15b5a3d6aee4-kube-api-access-hvfgx\") pod \"28dc2205-c86e-4bd3-b364-15b5a3d6aee4\" (UID: \"28dc2205-c86e-4bd3-b364-15b5a3d6aee4\") " Oct 30 13:42:48.209329 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.209305 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28dc2205-c86e-4bd3-b364-15b5a3d6aee4-plugins-config-volume" pod "28dc2205-c86e-4bd3-b364-15b5a3d6aee4" (UID: "28dc2205-c86e-4bd3-b364-15b5a3d6aee4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:48.209563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.209526 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b712a0-c94f-4e1d-b769-56b308d4e7b1-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:48.209563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.209543 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvbdh\" (UniqueName: \"kubernetes.io/projected/36b712a0-c94f-4e1d-b769-56b308d4e7b1-kube-api-access-pvbdh\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:48.209563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.209554 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dc2205-c86e-4bd3-b364-15b5a3d6aee4-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:48.211111 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.211089 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28dc2205-c86e-4bd3-b364-15b5a3d6aee4-kube-api-access-hvfgx" pod "28dc2205-c86e-4bd3-b364-15b5a3d6aee4" (UID: "28dc2205-c86e-4bd3-b364-15b5a3d6aee4"). InnerVolumeSpecName "kube-api-access-hvfgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:48.310635 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.310593 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvfgx\" (UniqueName: \"kubernetes.io/projected/28dc2205-c86e-4bd3-b364-15b5a3d6aee4-kube-api-access-hvfgx\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:48.428250 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.428197 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx"] Oct 30 13:42:48.432088 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:48.432059 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78dc89-d9466684-hdsmx"] Oct 30 13:42:49.089145 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:49.089114 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" Oct 30 13:42:49.089145 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:49.089130 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6" event={"ID":"28dc2205-c86e-4bd3-b364-15b5a3d6aee4","Type":"ContainerDied","Data":"986279fc018f01de2447216b81b860b04f83f7417ce10a0540e09149dffe9d04"} Oct 30 13:42:49.089762 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:49.089190 7010 scope.go:122] "RemoveContainer" containerID="fe6369a7f76008bf176028b11c1dc380090103006eb14882f40a13b52949794e" Oct 30 13:42:49.151037 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:49.151000 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6"] Oct 30 13:42:49.162324 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:49.162289 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78dc89-7bc476fc95-bfjl6"] Oct 30 13:42:49.491951 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:49.491913 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28dc2205-c86e-4bd3-b364-15b5a3d6aee4" path="/var/lib/kubelet/pods/28dc2205-c86e-4bd3-b364-15b5a3d6aee4/volumes" Oct 30 13:42:49.492371 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:49.492358 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b712a0-c94f-4e1d-b769-56b308d4e7b1" path="/var/lib/kubelet/pods/36b712a0-c94f-4e1d-b769-56b308d4e7b1/volumes" Oct 30 13:42:54.157849 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.157807 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c"] Oct 30 13:42:54.158719 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.158687 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" podUID="ca5ec0d2-744a-4cff-a863-4aae9666eccc" containerName="payload-processing" containerID="cri-o://c922d61609f3bfc2fb0d9ceefdb1689d93ad2f8f3e2163c0330041da09183c10" gracePeriod=30 Oct 30 13:42:54.191284 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.191247 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj"] Oct 30 13:42:54.191499 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.191466 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" podUID="d8dd1343-6d33-4c14-b861-0cc98485d047" containerName="payload-pre-processing" containerID="cri-o://bb93944c9306f7fc2196b03772b249c7594ff48caa8ae987bf7c8bc2397f2677" gracePeriod=30 Oct 30 13:42:54.663264 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.663235 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" Oct 30 13:42:54.673962 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.673927 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" Oct 30 13:42:54.740752 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.740628 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5ec0d2-744a-4cff-a863-4aae9666eccc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5ec0d2-744a-4cff-a863-4aae9666eccc-plugins-config-volume\") pod \"ca5ec0d2-744a-4cff-a863-4aae9666eccc\" (UID: \"ca5ec0d2-744a-4cff-a863-4aae9666eccc\") " Oct 30 13:42:54.740752 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.740727 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8dd1343-6d33-4c14-b861-0cc98485d047-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8dd1343-6d33-4c14-b861-0cc98485d047-plugins-config-volume\") pod \"d8dd1343-6d33-4c14-b861-0cc98485d047\" (UID: \"d8dd1343-6d33-4c14-b861-0cc98485d047\") " Oct 30 13:42:54.741001 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.740802 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5ec0d2-744a-4cff-a863-4aae9666eccc-kube-api-access-z8szp\" (UniqueName: \"kubernetes.io/projected/ca5ec0d2-744a-4cff-a863-4aae9666eccc-kube-api-access-z8szp\") pod \"ca5ec0d2-744a-4cff-a863-4aae9666eccc\" (UID: \"ca5ec0d2-744a-4cff-a863-4aae9666eccc\") " Oct 30 13:42:54.741001 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.740853 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8dd1343-6d33-4c14-b861-0cc98485d047-kube-api-access-68qq8\" (UniqueName: \"kubernetes.io/projected/d8dd1343-6d33-4c14-b861-0cc98485d047-kube-api-access-68qq8\") pod \"d8dd1343-6d33-4c14-b861-0cc98485d047\" (UID: \"d8dd1343-6d33-4c14-b861-0cc98485d047\") " Oct 30 13:42:54.741833 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.741775 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5ec0d2-744a-4cff-a863-4aae9666eccc-plugins-config-volume" pod "ca5ec0d2-744a-4cff-a863-4aae9666eccc" (UID: "ca5ec0d2-744a-4cff-a863-4aae9666eccc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:54.742232 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.742179 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8dd1343-6d33-4c14-b861-0cc98485d047-plugins-config-volume" pod "d8dd1343-6d33-4c14-b861-0cc98485d047" (UID: "d8dd1343-6d33-4c14-b861-0cc98485d047"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:42:54.744384 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.744360 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8dd1343-6d33-4c14-b861-0cc98485d047-kube-api-access-68qq8" pod "d8dd1343-6d33-4c14-b861-0cc98485d047" (UID: "d8dd1343-6d33-4c14-b861-0cc98485d047"). InnerVolumeSpecName "kube-api-access-68qq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:54.744832 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.744816 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5ec0d2-744a-4cff-a863-4aae9666eccc-kube-api-access-z8szp" pod "ca5ec0d2-744a-4cff-a863-4aae9666eccc" (UID: "ca5ec0d2-744a-4cff-a863-4aae9666eccc"). InnerVolumeSpecName "kube-api-access-z8szp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:42:54.843278 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.843235 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5ec0d2-744a-4cff-a863-4aae9666eccc-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.843278 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.843277 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8dd1343-6d33-4c14-b861-0cc98485d047-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.843516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.843292 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8szp\" (UniqueName: \"kubernetes.io/projected/ca5ec0d2-744a-4cff-a863-4aae9666eccc-kube-api-access-z8szp\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:54.843516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:54.843307 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68qq8\" (UniqueName: \"kubernetes.io/projected/d8dd1343-6d33-4c14-b861-0cc98485d047-kube-api-access-68qq8\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:42:55.135998 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.135962 7010 generic.go:356] "Generic (PLEG): container finished" podID="d8dd1343-6d33-4c14-b861-0cc98485d047" containerID="bb93944c9306f7fc2196b03772b249c7594ff48caa8ae987bf7c8bc2397f2677" exitCode=0 Oct 30 13:42:55.136209 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.136028 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" Oct 30 13:42:55.136209 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.136042 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" event={"ID":"d8dd1343-6d33-4c14-b861-0cc98485d047","Type":"ContainerDied","Data":"bb93944c9306f7fc2196b03772b249c7594ff48caa8ae987bf7c8bc2397f2677"} Oct 30 13:42:55.136209 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.136120 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj" event={"ID":"d8dd1343-6d33-4c14-b861-0cc98485d047","Type":"ContainerDied","Data":"ba3820284acfd3f97426a03c8f9e74def41c49805fccc20c20df83e56f55775d"} Oct 30 13:42:55.136209 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.136144 7010 scope.go:122] "RemoveContainer" containerID="bb93944c9306f7fc2196b03772b249c7594ff48caa8ae987bf7c8bc2397f2677" Oct 30 13:42:55.137722 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.137688 7010 generic.go:356] "Generic (PLEG): container finished" podID="ca5ec0d2-744a-4cff-a863-4aae9666eccc" containerID="c922d61609f3bfc2fb0d9ceefdb1689d93ad2f8f3e2163c0330041da09183c10" exitCode=0 Oct 30 13:42:55.137846 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.137738 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" Oct 30 13:42:55.137846 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.137741 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" event={"ID":"ca5ec0d2-744a-4cff-a863-4aae9666eccc","Type":"ContainerDied","Data":"c922d61609f3bfc2fb0d9ceefdb1689d93ad2f8f3e2163c0330041da09183c10"} Oct 30 13:42:55.137846 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.137780 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c" event={"ID":"ca5ec0d2-744a-4cff-a863-4aae9666eccc","Type":"ContainerDied","Data":"e2d36868a6c4272806958e5b84ff384e69aaa851b1b5a212a7ad1755bd4a6e1e"} Oct 30 13:42:55.147889 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.147870 7010 scope.go:122] "RemoveContainer" containerID="bb93944c9306f7fc2196b03772b249c7594ff48caa8ae987bf7c8bc2397f2677" Oct 30 13:42:55.148268 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:42:55.148242 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb93944c9306f7fc2196b03772b249c7594ff48caa8ae987bf7c8bc2397f2677\": container with ID starting with bb93944c9306f7fc2196b03772b249c7594ff48caa8ae987bf7c8bc2397f2677 not found: ID does not exist" containerID="bb93944c9306f7fc2196b03772b249c7594ff48caa8ae987bf7c8bc2397f2677" Oct 30 13:42:55.148347 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.148283 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb93944c9306f7fc2196b03772b249c7594ff48caa8ae987bf7c8bc2397f2677"} err="failed to get container status \"bb93944c9306f7fc2196b03772b249c7594ff48caa8ae987bf7c8bc2397f2677\": rpc error: code = NotFound desc = could not find container \"bb93944c9306f7fc2196b03772b249c7594ff48caa8ae987bf7c8bc2397f2677\": container with ID starting with bb93944c9306f7fc2196b03772b249c7594ff48caa8ae987bf7c8bc2397f2677 not found: ID does not exist" Oct 30 13:42:55.148347 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.148310 7010 scope.go:122] "RemoveContainer" containerID="c922d61609f3bfc2fb0d9ceefdb1689d93ad2f8f3e2163c0330041da09183c10" Oct 30 13:42:55.157942 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.157920 7010 scope.go:122] "RemoveContainer" containerID="c922d61609f3bfc2fb0d9ceefdb1689d93ad2f8f3e2163c0330041da09183c10" Oct 30 13:42:55.158233 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:42:55.158210 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c922d61609f3bfc2fb0d9ceefdb1689d93ad2f8f3e2163c0330041da09183c10\": container with ID starting with c922d61609f3bfc2fb0d9ceefdb1689d93ad2f8f3e2163c0330041da09183c10 not found: ID does not exist" containerID="c922d61609f3bfc2fb0d9ceefdb1689d93ad2f8f3e2163c0330041da09183c10" Oct 30 13:42:55.158275 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.158242 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c922d61609f3bfc2fb0d9ceefdb1689d93ad2f8f3e2163c0330041da09183c10"} err="failed to get container status \"c922d61609f3bfc2fb0d9ceefdb1689d93ad2f8f3e2163c0330041da09183c10\": rpc error: code = NotFound desc = could not find container \"c922d61609f3bfc2fb0d9ceefdb1689d93ad2f8f3e2163c0330041da09183c10\": container with ID starting with c922d61609f3bfc2fb0d9ceefdb1689d93ad2f8f3e2163c0330041da09183c10 not found: ID does not exist" Oct 30 13:42:55.195202 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.195161 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj"] Oct 30 13:42:55.207795 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.207759 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4fb25-5f9f89446c-fbvpj"] Oct 30 13:42:55.235007 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.234969 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c"] Oct 30 13:42:55.239448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.239420 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b4fb25-9bc7c6548-pts6c"] Oct 30 13:42:55.489062 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.488972 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5ec0d2-744a-4cff-a863-4aae9666eccc" path="/var/lib/kubelet/pods/ca5ec0d2-744a-4cff-a863-4aae9666eccc/volumes" Oct 30 13:42:55.489412 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:42:55.489398 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8dd1343-6d33-4c14-b861-0cc98485d047" path="/var/lib/kubelet/pods/d8dd1343-6d33-4c14-b861-0cc98485d047/volumes" Oct 30 13:42:59.401965 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:42:59.401929 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:43:25.485139 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:25.485103 7010 scope.go:122] "RemoveContainer" containerID="d01290c689d595f75173c06b7d9f72862637a2b170739cb9336574b93807225b" Oct 30 13:43:29.403285 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:43:29.403237 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:43:46.465389 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:46.465289 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb"] Oct 30 13:43:46.465945 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:46.465604 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" podUID="9e028b61-6039-4f07-b7a9-082d7b51c633" containerName="istio-proxy" containerID="cri-o://a683036c84a1166248026ae873ee53aa97c1355e9f7005a3814359a8045fa4b5" gracePeriod=30 Oct 30 13:43:46.488576 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:46.488533 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794"] Oct 30 13:43:46.488866 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:46.488813 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" podUID="1c94ab46-fc9f-4f4d-899c-423d65bde06e" containerName="istio-proxy" containerID="cri-o://87fe446182951d6c9801ff291312fa5827447e51e35324e8230fb77decc6fbb1" gracePeriod=30 Oct 30 13:43:50.003629 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:50.003582 7010 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:43:50.004034 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:50.003649 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" podUID="9e028b61-6039-4f07-b7a9-082d7b51c633" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:43:50.124858 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:50.124821 7010 patch_prober.go:29] interesting pod/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:43:50.125032 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:50.124884 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" podUID="1c94ab46-fc9f-4f4d-899c-423d65bde06e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:43:51.694603 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.694556 7010 generic.go:356] "Generic (PLEG): container finished" podID="1c94ab46-fc9f-4f4d-899c-423d65bde06e" containerID="87fe446182951d6c9801ff291312fa5827447e51e35324e8230fb77decc6fbb1" exitCode=0 Oct 30 13:43:51.695114 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.694821 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" event={"ID":"1c94ab46-fc9f-4f4d-899c-423d65bde06e","Type":"ContainerDied","Data":"87fe446182951d6c9801ff291312fa5827447e51e35324e8230fb77decc6fbb1"} Oct 30 13:43:51.697079 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.696956 7010 generic.go:356] "Generic (PLEG): container finished" podID="9e028b61-6039-4f07-b7a9-082d7b51c633" containerID="a683036c84a1166248026ae873ee53aa97c1355e9f7005a3814359a8045fa4b5" exitCode=0 Oct 30 13:43:51.697079 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.697009 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" event={"ID":"9e028b61-6039-4f07-b7a9-082d7b51c633","Type":"ContainerDied","Data":"a683036c84a1166248026ae873ee53aa97c1355e9f7005a3814359a8045fa4b5"} Oct 30 13:43:51.778570 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.778535 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:43:51.826539 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.825411 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:43:51.890908 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.890866 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e028b61-6039-4f07-b7a9-082d7b51c633-kube-api-access-6cnqs\" (UniqueName: \"kubernetes.io/projected/9e028b61-6039-4f07-b7a9-082d7b51c633-kube-api-access-6cnqs\") pod \"9e028b61-6039-4f07-b7a9-082d7b51c633\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " Oct 30 13:43:51.891082 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.890921 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-podinfo\") pod \"9e028b61-6039-4f07-b7a9-082d7b51c633\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " Oct 30 13:43:51.891082 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.890982 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-credential-socket\") pod \"9e028b61-6039-4f07-b7a9-082d7b51c633\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " Oct 30 13:43:51.891082 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.891015 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-workload-socket\") pod \"9e028b61-6039-4f07-b7a9-082d7b51c633\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " Oct 30 13:43:51.891082 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.891059 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-data\") pod \"9e028b61-6039-4f07-b7a9-082d7b51c633\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " Oct 30 13:43:51.891302 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.891134 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-workload-certs\") pod \"9e028b61-6039-4f07-b7a9-082d7b51c633\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " Oct 30 13:43:51.891373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.891333 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-envoy\") pod \"9e028b61-6039-4f07-b7a9-082d7b51c633\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " Oct 30 13:43:51.891373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.891320 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-credential-socket" pod "9e028b61-6039-4f07-b7a9-082d7b51c633" (UID: "9e028b61-6039-4f07-b7a9-082d7b51c633"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.891582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.891550 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e028b61-6039-4f07-b7a9-082d7b51c633-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e028b61-6039-4f07-b7a9-082d7b51c633-istiod-ca-cert\") pod \"9e028b61-6039-4f07-b7a9-082d7b51c633\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " Oct 30 13:43:51.891669 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.891611 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-workload-certs" pod "9e028b61-6039-4f07-b7a9-082d7b51c633" (UID: "9e028b61-6039-4f07-b7a9-082d7b51c633"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.891669 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.891630 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-token\" (UniqueName: \"kubernetes.io/projected/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-token\") pod \"9e028b61-6039-4f07-b7a9-082d7b51c633\" (UID: \"9e028b61-6039-4f07-b7a9-082d7b51c633\") " Oct 30 13:43:51.891794 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.891683 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-data" pod "9e028b61-6039-4f07-b7a9-082d7b51c633" (UID: "9e028b61-6039-4f07-b7a9-082d7b51c633"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.891794 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.891748 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-workload-socket" pod "9e028b61-6039-4f07-b7a9-082d7b51c633" (UID: "9e028b61-6039-4f07-b7a9-082d7b51c633"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.892016 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.891989 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e028b61-6039-4f07-b7a9-082d7b51c633-istiod-ca-cert" pod "9e028b61-6039-4f07-b7a9-082d7b51c633" (UID: "9e028b61-6039-4f07-b7a9-082d7b51c633"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:51.892395 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.892375 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.892455 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.892402 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.892455 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.892419 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.892455 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.892434 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.892455 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.892447 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e028b61-6039-4f07-b7a9-082d7b51c633-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.893930 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.893902 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-podinfo" pod "9e028b61-6039-4f07-b7a9-082d7b51c633" (UID: "9e028b61-6039-4f07-b7a9-082d7b51c633"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:43:51.894049 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.894003 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-envoy" pod "9e028b61-6039-4f07-b7a9-082d7b51c633" (UID: "9e028b61-6039-4f07-b7a9-082d7b51c633"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.894123 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.894050 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e028b61-6039-4f07-b7a9-082d7b51c633-kube-api-access-6cnqs" pod "9e028b61-6039-4f07-b7a9-082d7b51c633" (UID: "9e028b61-6039-4f07-b7a9-082d7b51c633"). InnerVolumeSpecName "kube-api-access-6cnqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:51.902207 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.902163 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-token" pod "9e028b61-6039-4f07-b7a9-082d7b51c633" (UID: "9e028b61-6039-4f07-b7a9-082d7b51c633"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:51.993476 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.993371 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-credential-socket\") pod \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " Oct 30 13:43:51.994060 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.993541 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istiod-ca-cert\") pod \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " Oct 30 13:43:51.994060 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.993578 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-data\") pod \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " Oct 30 13:43:51.994060 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.993618 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-workload-socket\") pod \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " Oct 30 13:43:51.994060 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.993673 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-podinfo\") pod \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " Oct 30 13:43:51.994060 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.993707 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-envoy\") pod \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " Oct 30 13:43:51.994060 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.993732 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-credential-socket" pod "1c94ab46-fc9f-4f4d-899c-423d65bde06e" (UID: "1c94ab46-fc9f-4f4d-899c-423d65bde06e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.994060 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.993779 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-workload-certs\") pod \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " Oct 30 13:43:51.994060 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.993842 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-token\" (UniqueName: \"kubernetes.io/projected/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-token\") pod \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " Oct 30 13:43:51.994060 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.993919 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c94ab46-fc9f-4f4d-899c-423d65bde06e-kube-api-access-7rtzd\" (UniqueName: \"kubernetes.io/projected/1c94ab46-fc9f-4f4d-899c-423d65bde06e-kube-api-access-7rtzd\") pod \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\" (UID: \"1c94ab46-fc9f-4f4d-899c-423d65bde06e\") " Oct 30 13:43:51.994060 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.993957 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-data" pod "1c94ab46-fc9f-4f4d-899c-423d65bde06e" (UID: "1c94ab46-fc9f-4f4d-899c-423d65bde06e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.994060 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.993975 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istiod-ca-cert" pod "1c94ab46-fc9f-4f4d-899c-423d65bde06e" (UID: "1c94ab46-fc9f-4f4d-899c-423d65bde06e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:43:51.994679 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.994168 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-workload-certs" pod "1c94ab46-fc9f-4f4d-899c-423d65bde06e" (UID: "1c94ab46-fc9f-4f4d-899c-423d65bde06e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.994679 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.994553 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-workload-socket" pod "1c94ab46-fc9f-4f4d-899c-423d65bde06e" (UID: "1c94ab46-fc9f-4f4d-899c-423d65bde06e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.994792 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.994738 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.994792 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.994762 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.994792 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.994777 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.994792 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.994790 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.995020 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.994804 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.995020 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.994818 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.995020 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.994831 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cnqs\" (UniqueName: \"kubernetes.io/projected/9e028b61-6039-4f07-b7a9-082d7b51c633-kube-api-access-6cnqs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.995020 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.994846 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.995020 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.994864 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e028b61-6039-4f07-b7a9-082d7b51c633-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:51.996372 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.996346 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-podinfo" pod "1c94ab46-fc9f-4f4d-899c-423d65bde06e" (UID: "1c94ab46-fc9f-4f4d-899c-423d65bde06e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:43:51.996482 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.996461 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c94ab46-fc9f-4f4d-899c-423d65bde06e-kube-api-access-7rtzd" pod "1c94ab46-fc9f-4f4d-899c-423d65bde06e" (UID: "1c94ab46-fc9f-4f4d-899c-423d65bde06e"). InnerVolumeSpecName "kube-api-access-7rtzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:51.996759 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.996735 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-envoy" pod "1c94ab46-fc9f-4f4d-899c-423d65bde06e" (UID: "1c94ab46-fc9f-4f4d-899c-423d65bde06e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:43:51.997067 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:51.997045 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-token" pod "1c94ab46-fc9f-4f4d-899c-423d65bde06e" (UID: "1c94ab46-fc9f-4f4d-899c-423d65bde06e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:43:52.096463 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:52.096428 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:52.096463 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:52.096459 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rtzd\" (UniqueName: \"kubernetes.io/projected/1c94ab46-fc9f-4f4d-899c-423d65bde06e-kube-api-access-7rtzd\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:52.096463 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:52.096471 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:52.096764 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:52.096480 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c94ab46-fc9f-4f4d-899c-423d65bde06e-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:43:52.705030 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:52.704993 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" Oct 30 13:43:52.705030 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:52.705008 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb" event={"ID":"9e028b61-6039-4f07-b7a9-082d7b51c633","Type":"ContainerDied","Data":"784a1beaa9255fec605e6ad6cabff0e62d6521ef21e06ad50d4afcd409348a34"} Oct 30 13:43:52.705629 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:52.705076 7010 scope.go:122] "RemoveContainer" containerID="a683036c84a1166248026ae873ee53aa97c1355e9f7005a3814359a8045fa4b5" Oct 30 13:43:52.706650 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:52.706619 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" event={"ID":"1c94ab46-fc9f-4f4d-899c-423d65bde06e","Type":"ContainerDied","Data":"324b23e841b727c948c86abd9a56023f515f7cba851b640dad1070a68a2747d7"} Oct 30 13:43:52.706760 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:52.706636 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794" Oct 30 13:43:52.719302 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:52.719281 7010 scope.go:122] "RemoveContainer" containerID="87fe446182951d6c9801ff291312fa5827447e51e35324e8230fb77decc6fbb1" Oct 30 13:43:52.775474 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:52.775376 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794"] Oct 30 13:43:52.787955 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:52.787922 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-l4794"] Oct 30 13:43:52.821034 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:52.820998 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb"] Oct 30 13:43:52.823276 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:52.823250 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8ee35e-openshift-default-7bfb8dbcd7-2pwxb"] Oct 30 13:43:53.490046 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:53.490006 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c94ab46-fc9f-4f4d-899c-423d65bde06e" path="/var/lib/kubelet/pods/1c94ab46-fc9f-4f4d-899c-423d65bde06e/volumes" Oct 30 13:43:53.490564 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:43:53.490549 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e028b61-6039-4f07-b7a9-082d7b51c633" path="/var/lib/kubelet/pods/9e028b61-6039-4f07-b7a9-082d7b51c633/volumes" Oct 30 13:43:59.418762 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:43:59.418719 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:44:00.942839 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:00.942796 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg"] Oct 30 13:44:00.943391 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:00.943079 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" podUID="e8430e34-fc9c-42de-82d5-b4d2ffd07dfc" containerName="maas-api" containerID="cri-o://b62978e278df850f59eec426bf3e0200747b8d5951f08c0fa49c8bc3714e9401" gracePeriod=30 Oct 30 13:44:01.323458 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.323429 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:44:01.527692 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.527601 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-kube-api-access-fv9pk\" (UniqueName: \"kubernetes.io/projected/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-kube-api-access-fv9pk\") pod \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\" (UID: \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\") " Oct 30 13:44:01.527692 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.527672 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-maas-api-tls\") pod \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\" (UID: \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\") " Oct 30 13:44:01.527941 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.527745 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-openshift-service-ca\") pod \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\" (UID: \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\") " Oct 30 13:44:01.528053 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.528029 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-metrics-certs\") pod \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\" (UID: \"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc\") " Oct 30 13:44:01.528165 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.528144 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-openshift-service-ca" pod "e8430e34-fc9c-42de-82d5-b4d2ffd07dfc" (UID: "e8430e34-fc9c-42de-82d5-b4d2ffd07dfc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:01.528567 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.528542 7010 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-openshift-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:01.529951 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.529927 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-maas-api-tls" pod "e8430e34-fc9c-42de-82d5-b4d2ffd07dfc" (UID: "e8430e34-fc9c-42de-82d5-b4d2ffd07dfc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:01.530463 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.530442 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-metrics-certs" pod "e8430e34-fc9c-42de-82d5-b4d2ffd07dfc" (UID: "e8430e34-fc9c-42de-82d5-b4d2ffd07dfc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:01.530554 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.530469 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-kube-api-access-fv9pk" pod "e8430e34-fc9c-42de-82d5-b4d2ffd07dfc" (UID: "e8430e34-fc9c-42de-82d5-b4d2ffd07dfc"). InnerVolumeSpecName "kube-api-access-fv9pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:01.629355 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.629316 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-maas-api-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:01.629355 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.629351 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:01.629355 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.629361 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv9pk\" (UniqueName: \"kubernetes.io/projected/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc-kube-api-access-fv9pk\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:01.777791 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.777694 7010 generic.go:356] "Generic (PLEG): container finished" podID="e8430e34-fc9c-42de-82d5-b4d2ffd07dfc" containerID="b62978e278df850f59eec426bf3e0200747b8d5951f08c0fa49c8bc3714e9401" exitCode=0 Oct 30 13:44:01.777791 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.777749 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" Oct 30 13:44:01.777791 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.777777 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" event={"ID":"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc","Type":"ContainerDied","Data":"b62978e278df850f59eec426bf3e0200747b8d5951f08c0fa49c8bc3714e9401"} Oct 30 13:44:01.778078 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.777841 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg" event={"ID":"e8430e34-fc9c-42de-82d5-b4d2ffd07dfc","Type":"ContainerDied","Data":"9963517cff3e752490b24de99174e871f6ced703c1d8f5430d5785e8528b810a"} Oct 30 13:44:01.778078 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.777872 7010 scope.go:122] "RemoveContainer" containerID="b62978e278df850f59eec426bf3e0200747b8d5951f08c0fa49c8bc3714e9401" Oct 30 13:44:01.789879 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.789856 7010 scope.go:122] "RemoveContainer" containerID="b62978e278df850f59eec426bf3e0200747b8d5951f08c0fa49c8bc3714e9401" Oct 30 13:44:01.790213 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:01.790193 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b62978e278df850f59eec426bf3e0200747b8d5951f08c0fa49c8bc3714e9401\": container with ID starting with b62978e278df850f59eec426bf3e0200747b8d5951f08c0fa49c8bc3714e9401 not found: ID does not exist" containerID="b62978e278df850f59eec426bf3e0200747b8d5951f08c0fa49c8bc3714e9401" Oct 30 13:44:01.790275 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.790224 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b62978e278df850f59eec426bf3e0200747b8d5951f08c0fa49c8bc3714e9401"} err="failed to get container status \"b62978e278df850f59eec426bf3e0200747b8d5951f08c0fa49c8bc3714e9401\": rpc error: code = NotFound desc = could not find container \"b62978e278df850f59eec426bf3e0200747b8d5951f08c0fa49c8bc3714e9401\": container with ID starting with b62978e278df850f59eec426bf3e0200747b8d5951f08c0fa49c8bc3714e9401 not found: ID does not exist" Oct 30 13:44:01.857563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.857519 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg"] Oct 30 13:44:01.871611 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:01.871574 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b0ea9-797fc7f4d-j9qvg"] Oct 30 13:44:03.488619 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:03.488584 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8430e34-fc9c-42de-82d5-b4d2ffd07dfc" path="/var/lib/kubelet/pods/e8430e34-fc9c-42de-82d5-b4d2ffd07dfc/volumes" Oct 30 13:44:12.382093 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.382055 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7"] Oct 30 13:44:12.382677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.382298 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" podUID="57a206b4-1557-4b43-9b49-f49f981334e9" containerName="payload-processing" containerID="cri-o://876cd05627f298b22e4f62dceecfae3f747f4cb0e435f32f4eb4c91e9d9f6d93" gracePeriod=30 Oct 30 13:44:12.390765 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.390732 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp"] Oct 30 13:44:12.390941 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.390921 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" podUID="c5ba48c8-ab14-4261-8529-eefa097249d4" containerName="payload-pre-processing" containerID="cri-o://2969761ec7a439b998e8833e48c7fd899253dfc11ca8e475414df2427130b1c5" gracePeriod=30 Oct 30 13:44:12.781844 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.781807 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" Oct 30 13:44:12.835953 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.835922 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" Oct 30 13:44:12.849218 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.849179 7010 generic.go:356] "Generic (PLEG): container finished" podID="c5ba48c8-ab14-4261-8529-eefa097249d4" containerID="2969761ec7a439b998e8833e48c7fd899253dfc11ca8e475414df2427130b1c5" exitCode=0 Oct 30 13:44:12.849415 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.849234 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" Oct 30 13:44:12.849415 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.849266 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" event={"ID":"c5ba48c8-ab14-4261-8529-eefa097249d4","Type":"ContainerDied","Data":"2969761ec7a439b998e8833e48c7fd899253dfc11ca8e475414df2427130b1c5"} Oct 30 13:44:12.849415 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.849330 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp" event={"ID":"c5ba48c8-ab14-4261-8529-eefa097249d4","Type":"ContainerDied","Data":"d6b79723435f0f3168228c1973fa1d0d920d3ee1525a1ccc3e1202fcb9810971"} Oct 30 13:44:12.849415 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.849360 7010 scope.go:122] "RemoveContainer" containerID="2969761ec7a439b998e8833e48c7fd899253dfc11ca8e475414df2427130b1c5" Oct 30 13:44:12.852442 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.852411 7010 generic.go:356] "Generic (PLEG): container finished" podID="57a206b4-1557-4b43-9b49-f49f981334e9" containerID="876cd05627f298b22e4f62dceecfae3f747f4cb0e435f32f4eb4c91e9d9f6d93" exitCode=0 Oct 30 13:44:12.852609 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.852560 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" event={"ID":"57a206b4-1557-4b43-9b49-f49f981334e9","Type":"ContainerDied","Data":"876cd05627f298b22e4f62dceecfae3f747f4cb0e435f32f4eb4c91e9d9f6d93"} Oct 30 13:44:12.852687 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.852608 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" event={"ID":"57a206b4-1557-4b43-9b49-f49f981334e9","Type":"ContainerDied","Data":"119d633ba069e55bbcb66276bacffb4bf21c36e91766e0369db7bcd45da49475"} Oct 30 13:44:12.852687 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.852680 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7" Oct 30 13:44:12.862611 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.862585 7010 scope.go:122] "RemoveContainer" containerID="2969761ec7a439b998e8833e48c7fd899253dfc11ca8e475414df2427130b1c5" Oct 30 13:44:12.863013 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:12.862991 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2969761ec7a439b998e8833e48c7fd899253dfc11ca8e475414df2427130b1c5\": container with ID starting with 2969761ec7a439b998e8833e48c7fd899253dfc11ca8e475414df2427130b1c5 not found: ID does not exist" containerID="2969761ec7a439b998e8833e48c7fd899253dfc11ca8e475414df2427130b1c5" Oct 30 13:44:12.863115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.863028 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2969761ec7a439b998e8833e48c7fd899253dfc11ca8e475414df2427130b1c5"} err="failed to get container status \"2969761ec7a439b998e8833e48c7fd899253dfc11ca8e475414df2427130b1c5\": rpc error: code = NotFound desc = could not find container \"2969761ec7a439b998e8833e48c7fd899253dfc11ca8e475414df2427130b1c5\": container with ID starting with 2969761ec7a439b998e8833e48c7fd899253dfc11ca8e475414df2427130b1c5 not found: ID does not exist" Oct 30 13:44:12.863115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.863054 7010 scope.go:122] "RemoveContainer" containerID="876cd05627f298b22e4f62dceecfae3f747f4cb0e435f32f4eb4c91e9d9f6d93" Oct 30 13:44:12.874041 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.874014 7010 scope.go:122] "RemoveContainer" containerID="876cd05627f298b22e4f62dceecfae3f747f4cb0e435f32f4eb4c91e9d9f6d93" Oct 30 13:44:12.874377 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:12.874354 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"876cd05627f298b22e4f62dceecfae3f747f4cb0e435f32f4eb4c91e9d9f6d93\": container with ID starting with 876cd05627f298b22e4f62dceecfae3f747f4cb0e435f32f4eb4c91e9d9f6d93 not found: ID does not exist" containerID="876cd05627f298b22e4f62dceecfae3f747f4cb0e435f32f4eb4c91e9d9f6d93" Oct 30 13:44:12.874454 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.874388 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876cd05627f298b22e4f62dceecfae3f747f4cb0e435f32f4eb4c91e9d9f6d93"} err="failed to get container status \"876cd05627f298b22e4f62dceecfae3f747f4cb0e435f32f4eb4c91e9d9f6d93\": rpc error: code = NotFound desc = could not find container \"876cd05627f298b22e4f62dceecfae3f747f4cb0e435f32f4eb4c91e9d9f6d93\": container with ID starting with 876cd05627f298b22e4f62dceecfae3f747f4cb0e435f32f4eb4c91e9d9f6d93 not found: ID does not exist" Oct 30 13:44:12.926520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.926405 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ba48c8-ab14-4261-8529-eefa097249d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ba48c8-ab14-4261-8529-eefa097249d4-plugins-config-volume\") pod \"c5ba48c8-ab14-4261-8529-eefa097249d4\" (UID: \"c5ba48c8-ab14-4261-8529-eefa097249d4\") " Oct 30 13:44:12.926520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.926459 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a206b4-1557-4b43-9b49-f49f981334e9-kube-api-access-4zg7h\" (UniqueName: \"kubernetes.io/projected/57a206b4-1557-4b43-9b49-f49f981334e9-kube-api-access-4zg7h\") pod \"57a206b4-1557-4b43-9b49-f49f981334e9\" (UID: \"57a206b4-1557-4b43-9b49-f49f981334e9\") " Oct 30 13:44:12.926776 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.926546 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a206b4-1557-4b43-9b49-f49f981334e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a206b4-1557-4b43-9b49-f49f981334e9-plugins-config-volume\") pod \"57a206b4-1557-4b43-9b49-f49f981334e9\" (UID: \"57a206b4-1557-4b43-9b49-f49f981334e9\") " Oct 30 13:44:12.926841 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.926781 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ba48c8-ab14-4261-8529-eefa097249d4-kube-api-access-87bf8\" (UniqueName: \"kubernetes.io/projected/c5ba48c8-ab14-4261-8529-eefa097249d4-kube-api-access-87bf8\") pod \"c5ba48c8-ab14-4261-8529-eefa097249d4\" (UID: \"c5ba48c8-ab14-4261-8529-eefa097249d4\") " Oct 30 13:44:12.926914 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.926853 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ba48c8-ab14-4261-8529-eefa097249d4-plugins-config-volume" pod "c5ba48c8-ab14-4261-8529-eefa097249d4" (UID: "c5ba48c8-ab14-4261-8529-eefa097249d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:12.927079 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.927013 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a206b4-1557-4b43-9b49-f49f981334e9-plugins-config-volume" pod "57a206b4-1557-4b43-9b49-f49f981334e9" (UID: "57a206b4-1557-4b43-9b49-f49f981334e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:12.927426 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.927404 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ba48c8-ab14-4261-8529-eefa097249d4-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:12.927518 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.927430 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a206b4-1557-4b43-9b49-f49f981334e9-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:12.928791 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.928766 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a206b4-1557-4b43-9b49-f49f981334e9-kube-api-access-4zg7h" pod "57a206b4-1557-4b43-9b49-f49f981334e9" (UID: "57a206b4-1557-4b43-9b49-f49f981334e9"). InnerVolumeSpecName "kube-api-access-4zg7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:12.928968 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:12.928951 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ba48c8-ab14-4261-8529-eefa097249d4-kube-api-access-87bf8" pod "c5ba48c8-ab14-4261-8529-eefa097249d4" (UID: "c5ba48c8-ab14-4261-8529-eefa097249d4"). InnerVolumeSpecName "kube-api-access-87bf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:13.029176 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:13.029136 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87bf8\" (UniqueName: \"kubernetes.io/projected/c5ba48c8-ab14-4261-8529-eefa097249d4-kube-api-access-87bf8\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:13.029176 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:13.029173 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zg7h\" (UniqueName: \"kubernetes.io/projected/57a206b4-1557-4b43-9b49-f49f981334e9-kube-api-access-4zg7h\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:13.235947 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:13.235857 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp"] Oct 30 13:44:13.246286 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:13.246249 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fca422-57d9486bcf-vlcfp"] Oct 30 13:44:13.282337 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:13.282298 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7"] Oct 30 13:44:13.287990 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:13.287960 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fca422-56b46c4d56-mpcp7"] Oct 30 13:44:13.488565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:13.488468 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57a206b4-1557-4b43-9b49-f49f981334e9" path="/var/lib/kubelet/pods/57a206b4-1557-4b43-9b49-f49f981334e9/volumes" Oct 30 13:44:13.488970 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:13.488954 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ba48c8-ab14-4261-8529-eefa097249d4" path="/var/lib/kubelet/pods/c5ba48c8-ab14-4261-8529-eefa097249d4/volumes" Oct 30 13:44:14.979169 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:14.979127 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh"] Oct 30 13:44:15.010919 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.010879 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq"] Oct 30 13:44:15.068303 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.068265 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk"] Oct 30 13:44:15.068623 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.068564 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" podUID="998f2069-f1df-403c-9719-46a9a3cda3fa" containerName="maas-api" containerID="cri-o://bb8db0c1c99387b9a10741c73f5a99a73a73b8a70fc2504738274b80bdc4c34e" gracePeriod=30 Oct 30 13:44:15.449206 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.449167 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:44:15.564785 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.564687 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/998f2069-f1df-403c-9719-46a9a3cda3fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/998f2069-f1df-403c-9719-46a9a3cda3fa-maas-api-tls\") pod \"998f2069-f1df-403c-9719-46a9a3cda3fa\" (UID: \"998f2069-f1df-403c-9719-46a9a3cda3fa\") " Oct 30 13:44:15.564973 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.564832 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998f2069-f1df-403c-9719-46a9a3cda3fa-kube-api-access-nz2s8\" (UniqueName: \"kubernetes.io/projected/998f2069-f1df-403c-9719-46a9a3cda3fa-kube-api-access-nz2s8\") pod \"998f2069-f1df-403c-9719-46a9a3cda3fa\" (UID: \"998f2069-f1df-403c-9719-46a9a3cda3fa\") " Oct 30 13:44:15.564973 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.564954 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998f2069-f1df-403c-9719-46a9a3cda3fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/998f2069-f1df-403c-9719-46a9a3cda3fa-openshift-service-ca\") pod \"998f2069-f1df-403c-9719-46a9a3cda3fa\" (UID: \"998f2069-f1df-403c-9719-46a9a3cda3fa\") " Oct 30 13:44:15.565112 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.565013 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/998f2069-f1df-403c-9719-46a9a3cda3fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/998f2069-f1df-403c-9719-46a9a3cda3fa-metrics-certs\") pod \"998f2069-f1df-403c-9719-46a9a3cda3fa\" (UID: \"998f2069-f1df-403c-9719-46a9a3cda3fa\") " Oct 30 13:44:15.565368 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.565335 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998f2069-f1df-403c-9719-46a9a3cda3fa-openshift-service-ca" pod "998f2069-f1df-403c-9719-46a9a3cda3fa" (UID: "998f2069-f1df-403c-9719-46a9a3cda3fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:15.565927 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.565647 7010 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/998f2069-f1df-403c-9719-46a9a3cda3fa-openshift-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:15.567273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.567245 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998f2069-f1df-403c-9719-46a9a3cda3fa-kube-api-access-nz2s8" pod "998f2069-f1df-403c-9719-46a9a3cda3fa" (UID: "998f2069-f1df-403c-9719-46a9a3cda3fa"). InnerVolumeSpecName "kube-api-access-nz2s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:15.567379 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.567328 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/998f2069-f1df-403c-9719-46a9a3cda3fa-maas-api-tls" pod "998f2069-f1df-403c-9719-46a9a3cda3fa" (UID: "998f2069-f1df-403c-9719-46a9a3cda3fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:15.567379 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.567347 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/998f2069-f1df-403c-9719-46a9a3cda3fa-metrics-certs" pod "998f2069-f1df-403c-9719-46a9a3cda3fa" (UID: "998f2069-f1df-403c-9719-46a9a3cda3fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:15.674512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.673956 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz2s8\" (UniqueName: \"kubernetes.io/projected/998f2069-f1df-403c-9719-46a9a3cda3fa-kube-api-access-nz2s8\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:15.674512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.673994 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/998f2069-f1df-403c-9719-46a9a3cda3fa-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:15.674512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.674012 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/998f2069-f1df-403c-9719-46a9a3cda3fa-maas-api-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:15.874321 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.874285 7010 generic.go:356] "Generic (PLEG): container finished" podID="998f2069-f1df-403c-9719-46a9a3cda3fa" containerID="bb8db0c1c99387b9a10741c73f5a99a73a73b8a70fc2504738274b80bdc4c34e" exitCode=0 Oct 30 13:44:15.874587 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.874330 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" Oct 30 13:44:15.874587 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.874355 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" event={"ID":"998f2069-f1df-403c-9719-46a9a3cda3fa","Type":"ContainerDied","Data":"bb8db0c1c99387b9a10741c73f5a99a73a73b8a70fc2504738274b80bdc4c34e"} Oct 30 13:44:15.874587 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.874409 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk" event={"ID":"998f2069-f1df-403c-9719-46a9a3cda3fa","Type":"ContainerDied","Data":"28ea46e187b0872a540d5931fa71c05f19be485462b3d323bbac3f796ed3639e"} Oct 30 13:44:15.874587 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.874438 7010 scope.go:122] "RemoveContainer" containerID="bb8db0c1c99387b9a10741c73f5a99a73a73b8a70fc2504738274b80bdc4c34e" Oct 30 13:44:15.887343 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.887319 7010 scope.go:122] "RemoveContainer" containerID="bb8db0c1c99387b9a10741c73f5a99a73a73b8a70fc2504738274b80bdc4c34e" Oct 30 13:44:15.887729 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:15.887704 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb8db0c1c99387b9a10741c73f5a99a73a73b8a70fc2504738274b80bdc4c34e\": container with ID starting with bb8db0c1c99387b9a10741c73f5a99a73a73b8a70fc2504738274b80bdc4c34e not found: ID does not exist" containerID="bb8db0c1c99387b9a10741c73f5a99a73a73b8a70fc2504738274b80bdc4c34e" Oct 30 13:44:15.887819 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.887742 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8db0c1c99387b9a10741c73f5a99a73a73b8a70fc2504738274b80bdc4c34e"} err="failed to get container status \"bb8db0c1c99387b9a10741c73f5a99a73a73b8a70fc2504738274b80bdc4c34e\": rpc error: code = NotFound desc = could not find container \"bb8db0c1c99387b9a10741c73f5a99a73a73b8a70fc2504738274b80bdc4c34e\": container with ID starting with bb8db0c1c99387b9a10741c73f5a99a73a73b8a70fc2504738274b80bdc4c34e not found: ID does not exist" Oct 30 13:44:15.933623 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.933590 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk"] Oct 30 13:44:15.947036 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:15.946998 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fdd4d4-6c5cf7c8c5-cxbsk"] Oct 30 13:44:17.490773 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:17.490732 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998f2069-f1df-403c-9719-46a9a3cda3fa" path="/var/lib/kubelet/pods/998f2069-f1df-403c-9719-46a9a3cda3fa/volumes" Oct 30 13:44:18.385741 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:18.385697 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd"] Oct 30 13:44:18.386027 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:18.385977 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" podUID="415237a2-07c0-481d-bb7b-f818c83e92ce" containerName="istio-proxy" containerID="cri-o://1ec1e6006a2769811c0ca7d781ef431f29b11040a8508bb46bdfc7e06cb91fca" gracePeriod=30 Oct 30 13:44:23.677711 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.677683 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:44:23.698867 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.698833 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-workload-certs\") pod \"415237a2-07c0-481d-bb7b-f818c83e92ce\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " Oct 30 13:44:23.699085 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.698910 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-podinfo\") pod \"415237a2-07c0-481d-bb7b-f818c83e92ce\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " Oct 30 13:44:23.699085 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.698983 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415237a2-07c0-481d-bb7b-f818c83e92ce-kube-api-access-7qbrm\" (UniqueName: \"kubernetes.io/projected/415237a2-07c0-481d-bb7b-f818c83e92ce-kube-api-access-7qbrm\") pod \"415237a2-07c0-481d-bb7b-f818c83e92ce\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " Oct 30 13:44:23.699085 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.699018 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-workload-socket\") pod \"415237a2-07c0-481d-bb7b-f818c83e92ce\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " Oct 30 13:44:23.699085 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.699046 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-envoy\") pod \"415237a2-07c0-481d-bb7b-f818c83e92ce\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " Oct 30 13:44:23.699311 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.699127 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-workload-certs" pod "415237a2-07c0-481d-bb7b-f818c83e92ce" (UID: "415237a2-07c0-481d-bb7b-f818c83e92ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:23.699377 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.699315 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-workload-socket" pod "415237a2-07c0-481d-bb7b-f818c83e92ce" (UID: "415237a2-07c0-481d-bb7b-f818c83e92ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:23.699377 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.699334 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-credential-socket\") pod \"415237a2-07c0-481d-bb7b-f818c83e92ce\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " Oct 30 13:44:23.699504 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.699418 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-data\") pod \"415237a2-07c0-481d-bb7b-f818c83e92ce\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " Oct 30 13:44:23.699564 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.699532 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-token\" (UniqueName: \"kubernetes.io/projected/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-token\") pod \"415237a2-07c0-481d-bb7b-f818c83e92ce\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " Oct 30 13:44:23.699677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.699553 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-credential-socket" pod "415237a2-07c0-481d-bb7b-f818c83e92ce" (UID: "415237a2-07c0-481d-bb7b-f818c83e92ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:23.699677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.699576 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/415237a2-07c0-481d-bb7b-f818c83e92ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415237a2-07c0-481d-bb7b-f818c83e92ce-istiod-ca-cert\") pod \"415237a2-07c0-481d-bb7b-f818c83e92ce\" (UID: \"415237a2-07c0-481d-bb7b-f818c83e92ce\") " Oct 30 13:44:23.699794 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.699683 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-data" pod "415237a2-07c0-481d-bb7b-f818c83e92ce" (UID: "415237a2-07c0-481d-bb7b-f818c83e92ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:23.700072 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.700049 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415237a2-07c0-481d-bb7b-f818c83e92ce-istiod-ca-cert" pod "415237a2-07c0-481d-bb7b-f818c83e92ce" (UID: "415237a2-07c0-481d-bb7b-f818c83e92ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:23.700243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.700217 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:23.700334 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.700248 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:23.700334 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.700267 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:23.700334 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.700288 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415237a2-07c0-481d-bb7b-f818c83e92ce-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:23.700334 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.700301 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:23.701556 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.701534 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-envoy" pod "415237a2-07c0-481d-bb7b-f818c83e92ce" (UID: "415237a2-07c0-481d-bb7b-f818c83e92ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:23.701841 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.701820 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-token" pod "415237a2-07c0-481d-bb7b-f818c83e92ce" (UID: "415237a2-07c0-481d-bb7b-f818c83e92ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:23.701897 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.701884 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-podinfo" pod "415237a2-07c0-481d-bb7b-f818c83e92ce" (UID: "415237a2-07c0-481d-bb7b-f818c83e92ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:44:23.702034 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.702020 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415237a2-07c0-481d-bb7b-f818c83e92ce-kube-api-access-7qbrm" pod "415237a2-07c0-481d-bb7b-f818c83e92ce" (UID: "415237a2-07c0-481d-bb7b-f818c83e92ce"). InnerVolumeSpecName "kube-api-access-7qbrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:23.801996 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.801953 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qbrm\" (UniqueName: \"kubernetes.io/projected/415237a2-07c0-481d-bb7b-f818c83e92ce-kube-api-access-7qbrm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:23.801996 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.801996 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:23.802267 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.802011 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:23.802267 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.802025 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415237a2-07c0-481d-bb7b-f818c83e92ce-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:23.932566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.932438 7010 generic.go:356] "Generic (PLEG): container finished" podID="415237a2-07c0-481d-bb7b-f818c83e92ce" containerID="1ec1e6006a2769811c0ca7d781ef431f29b11040a8508bb46bdfc7e06cb91fca" exitCode=0 Oct 30 13:44:23.932566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.932530 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" Oct 30 13:44:23.932566 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.932529 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" event={"ID":"415237a2-07c0-481d-bb7b-f818c83e92ce","Type":"ContainerDied","Data":"1ec1e6006a2769811c0ca7d781ef431f29b11040a8508bb46bdfc7e06cb91fca"} Oct 30 13:44:23.932872 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.932591 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd" event={"ID":"415237a2-07c0-481d-bb7b-f818c83e92ce","Type":"ContainerDied","Data":"df0c1ac3c94e8e70e239fd4bbcaf7f6963798fb310975018de7cfa528dbccc2c"} Oct 30 13:44:23.932872 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.932620 7010 scope.go:122] "RemoveContainer" containerID="1ec1e6006a2769811c0ca7d781ef431f29b11040a8508bb46bdfc7e06cb91fca" Oct 30 13:44:23.943813 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.943790 7010 scope.go:122] "RemoveContainer" containerID="00061988993a7660d7b7ff0f9c51ee66c0c4a28107ff661c82543a4198ffe954" Oct 30 13:44:23.954634 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.954607 7010 scope.go:122] "RemoveContainer" containerID="1ec1e6006a2769811c0ca7d781ef431f29b11040a8508bb46bdfc7e06cb91fca" Oct 30 13:44:23.954966 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:23.954946 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec1e6006a2769811c0ca7d781ef431f29b11040a8508bb46bdfc7e06cb91fca\": container with ID starting with 1ec1e6006a2769811c0ca7d781ef431f29b11040a8508bb46bdfc7e06cb91fca not found: ID does not exist" containerID="1ec1e6006a2769811c0ca7d781ef431f29b11040a8508bb46bdfc7e06cb91fca" Oct 30 13:44:23.955040 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.954976 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec1e6006a2769811c0ca7d781ef431f29b11040a8508bb46bdfc7e06cb91fca"} err="failed to get container status \"1ec1e6006a2769811c0ca7d781ef431f29b11040a8508bb46bdfc7e06cb91fca\": rpc error: code = NotFound desc = could not find container \"1ec1e6006a2769811c0ca7d781ef431f29b11040a8508bb46bdfc7e06cb91fca\": container with ID starting with 1ec1e6006a2769811c0ca7d781ef431f29b11040a8508bb46bdfc7e06cb91fca not found: ID does not exist" Oct 30 13:44:23.955040 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.955004 7010 scope.go:122] "RemoveContainer" containerID="00061988993a7660d7b7ff0f9c51ee66c0c4a28107ff661c82543a4198ffe954" Oct 30 13:44:23.955332 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:23.955309 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00061988993a7660d7b7ff0f9c51ee66c0c4a28107ff661c82543a4198ffe954\": container with ID starting with 00061988993a7660d7b7ff0f9c51ee66c0c4a28107ff661c82543a4198ffe954 not found: ID does not exist" containerID="00061988993a7660d7b7ff0f9c51ee66c0c4a28107ff661c82543a4198ffe954" Oct 30 13:44:23.955393 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.955338 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00061988993a7660d7b7ff0f9c51ee66c0c4a28107ff661c82543a4198ffe954"} err="failed to get container status \"00061988993a7660d7b7ff0f9c51ee66c0c4a28107ff661c82543a4198ffe954\": rpc error: code = NotFound desc = could not find container \"00061988993a7660d7b7ff0f9c51ee66c0c4a28107ff661c82543a4198ffe954\": container with ID starting with 00061988993a7660d7b7ff0f9c51ee66c0c4a28107ff661c82543a4198ffe954 not found: ID does not exist" Oct 30 13:44:23.995234 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:23.995192 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd"] Oct 30 13:44:24.002793 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:24.002757 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9cbcbc-openshift-default-78d89f8497-2k9qd"] Oct 30 13:44:25.060727 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:25.060679 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw"] Oct 30 13:44:25.061231 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:25.060951 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" podUID="fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" containerName="istio-proxy" containerID="cri-o://ce05764bdeb208538cf70152a554d5136ccb8828f2a2777a225919e99ab6725a" gracePeriod=30 Oct 30 13:44:25.488884 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:25.488851 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="415237a2-07c0-481d-bb7b-f818c83e92ce" path="/var/lib/kubelet/pods/415237a2-07c0-481d-bb7b-f818c83e92ce/volumes" Oct 30 13:44:25.578239 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:25.578197 7010 scope.go:122] "RemoveContainer" containerID="97963a5067a4a9242a1dc64d5abb9327cdcc432569eb0ed68778e71c7399a0d3" Oct 30 13:44:27.009938 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:27.009893 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:44:27.010434 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:27.009974 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" podUID="fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:44:27.850965 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:27.850927 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s"] Oct 30 13:44:27.851188 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:27.851142 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" podUID="5dafc909-0a2f-41dd-9529-76cbfad4ce5f" containerName="istio-proxy" containerID="cri-o://604ff51449344ccc20f224a52bc1b80c8a0919729e934939b300619299c20086" gracePeriod=30 Oct 30 13:44:28.335819 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:28.335772 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:44:28.336253 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:28.335847 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" podUID="5dafc909-0a2f-41dd-9529-76cbfad4ce5f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:44:29.402587 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:29.402530 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:44:30.025764 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.025700 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" podUID="a14c1b45-7310-4620-bc17-07354849809e" containerName="main" containerID="cri-o://27d7d6d97303afbd413eb046a4d4f8625dcfdb1df030ec554d658771d9db3db8" gracePeriod=45 Oct 30 13:44:30.035196 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.035004 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" containerID="cri-o://8e6091a0468e8188798e496ee01e4568f3c4077453ff8f28603063277c5a711e" gracePeriod=45 Oct 30 13:44:30.348226 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.348185 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:44:30.366466 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.366435 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:44:30.381623 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.381592 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:44:30.387793 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.387761 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3640a4c-2e41-4186-ae55-d7a03c986793-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3640a4c-2e41-4186-ae55-d7a03c986793-tls-certs\") pod \"b3640a4c-2e41-4186-ae55-d7a03c986793\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " Oct 30 13:44:30.388046 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.387811 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-tmp-dir\") pod \"a14c1b45-7310-4620-bc17-07354849809e\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " Oct 30 13:44:30.388046 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.387841 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-home\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-home\") pod \"b3640a4c-2e41-4186-ae55-d7a03c986793\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " Oct 30 13:44:30.388046 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.387861 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-dshm\") pod \"a14c1b45-7310-4620-bc17-07354849809e\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " Oct 30 13:44:30.388046 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.387882 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-model-cache\") pod \"b3640a4c-2e41-4186-ae55-d7a03c986793\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " Oct 30 13:44:30.388046 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.387907 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-model-cache\") pod \"a14c1b45-7310-4620-bc17-07354849809e\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " Oct 30 13:44:30.388046 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.387960 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3640a4c-2e41-4186-ae55-d7a03c986793-kube-api-access-pzwts\" (UniqueName: \"kubernetes.io/projected/b3640a4c-2e41-4186-ae55-d7a03c986793-kube-api-access-pzwts\") pod \"b3640a4c-2e41-4186-ae55-d7a03c986793\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " Oct 30 13:44:30.388046 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.387996 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-tmp-dir\") pod \"b3640a4c-2e41-4186-ae55-d7a03c986793\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " Oct 30 13:44:30.388464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.388051 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a14c1b45-7310-4620-bc17-07354849809e-kube-api-access-8vzkr\" (UniqueName: \"kubernetes.io/projected/a14c1b45-7310-4620-bc17-07354849809e-kube-api-access-8vzkr\") pod \"a14c1b45-7310-4620-bc17-07354849809e\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " Oct 30 13:44:30.388464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.388174 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-home" pod "b3640a4c-2e41-4186-ae55-d7a03c986793" (UID: "b3640a4c-2e41-4186-ae55-d7a03c986793"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:30.388464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.388183 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a14c1b45-7310-4620-bc17-07354849809e-tls-certs\" (UniqueName: \"kubernetes.io/secret/a14c1b45-7310-4620-bc17-07354849809e-tls-certs\") pod \"a14c1b45-7310-4620-bc17-07354849809e\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " Oct 30 13:44:30.388464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.388183 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-tmp-dir" pod "a14c1b45-7310-4620-bc17-07354849809e" (UID: "a14c1b45-7310-4620-bc17-07354849809e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:30.388464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.388189 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-model-cache" pod "b3640a4c-2e41-4186-ae55-d7a03c986793" (UID: "b3640a4c-2e41-4186-ae55-d7a03c986793"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:30.388464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.388217 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-dshm\") pod \"b3640a4c-2e41-4186-ae55-d7a03c986793\" (UID: \"b3640a4c-2e41-4186-ae55-d7a03c986793\") " Oct 30 13:44:30.388464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.388333 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-home\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-home\") pod \"a14c1b45-7310-4620-bc17-07354849809e\" (UID: \"a14c1b45-7310-4620-bc17-07354849809e\") " Oct 30 13:44:30.388464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.388396 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-tmp-dir" pod "b3640a4c-2e41-4186-ae55-d7a03c986793" (UID: "b3640a4c-2e41-4186-ae55-d7a03c986793"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:30.388958 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.388412 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-model-cache" pod "a14c1b45-7310-4620-bc17-07354849809e" (UID: "a14c1b45-7310-4620-bc17-07354849809e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:30.388958 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.388767 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-home" pod "a14c1b45-7310-4620-bc17-07354849809e" (UID: "a14c1b45-7310-4620-bc17-07354849809e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:30.389076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.388961 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.389076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.388985 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.389076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.389001 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.389076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.389014 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.389076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.389027 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.389076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.389040 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.390885 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.390769 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-dshm" pod "b3640a4c-2e41-4186-ae55-d7a03c986793" (UID: "b3640a4c-2e41-4186-ae55-d7a03c986793"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:30.390885 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.390778 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-dshm" pod "a14c1b45-7310-4620-bc17-07354849809e" (UID: "a14c1b45-7310-4620-bc17-07354849809e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:30.390885 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.390793 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14c1b45-7310-4620-bc17-07354849809e-kube-api-access-8vzkr" pod "a14c1b45-7310-4620-bc17-07354849809e" (UID: "a14c1b45-7310-4620-bc17-07354849809e"). InnerVolumeSpecName "kube-api-access-8vzkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:30.390885 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.390855 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a14c1b45-7310-4620-bc17-07354849809e-tls-certs" pod "a14c1b45-7310-4620-bc17-07354849809e" (UID: "a14c1b45-7310-4620-bc17-07354849809e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:30.390885 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.390868 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3640a4c-2e41-4186-ae55-d7a03c986793-tls-certs" pod "b3640a4c-2e41-4186-ae55-d7a03c986793" (UID: "b3640a4c-2e41-4186-ae55-d7a03c986793"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:30.391161 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.391122 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3640a4c-2e41-4186-ae55-d7a03c986793-kube-api-access-pzwts" pod "b3640a4c-2e41-4186-ae55-d7a03c986793" (UID: "b3640a4c-2e41-4186-ae55-d7a03c986793"). InnerVolumeSpecName "kube-api-access-pzwts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:30.490371 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.490329 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-token\" (UniqueName: \"kubernetes.io/projected/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-token\") pod \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " Oct 30 13:44:30.491418 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.490429 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-credential-socket\") pod \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " Oct 30 13:44:30.491418 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.490461 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-kube-api-access-btz5g\" (UniqueName: \"kubernetes.io/projected/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-kube-api-access-btz5g\") pod \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " Oct 30 13:44:30.491418 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.490542 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-workload-certs\") pod \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " Oct 30 13:44:30.491418 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.490598 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-envoy\") pod \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " Oct 30 13:44:30.491418 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.490635 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-data\") pod \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " Oct 30 13:44:30.491418 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.490680 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-podinfo\") pod \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " Oct 30 13:44:30.491418 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.490759 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-workload-socket\") pod \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " Oct 30 13:44:30.491418 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.490808 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istiod-ca-cert\") pod \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\" (UID: \"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231\") " Oct 30 13:44:30.491418 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.490989 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-credential-socket" pod "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" (UID: "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:30.491975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.491422 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istiod-ca-cert" pod "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" (UID: "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:30.491975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.491441 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-data" pod "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" (UID: "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:30.491975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.491547 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14c1b45-7310-4620-bc17-07354849809e-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.491975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.491568 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3640a4c-2e41-4186-ae55-d7a03c986793-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.491975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.491582 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.491975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.491598 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3640a4c-2e41-4186-ae55-d7a03c986793-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.491975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.491612 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14c1b45-7310-4620-bc17-07354849809e-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.491975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.491626 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzwts\" (UniqueName: \"kubernetes.io/projected/b3640a4c-2e41-4186-ae55-d7a03c986793-kube-api-access-pzwts\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.491975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.491641 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vzkr\" (UniqueName: \"kubernetes.io/projected/a14c1b45-7310-4620-bc17-07354849809e-kube-api-access-8vzkr\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.491975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.491650 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-workload-socket" pod "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" (UID: "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:30.491975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.491476 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-workload-certs" pod "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" (UID: "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:30.493423 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.493393 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-envoy" pod "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" (UID: "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:30.493739 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.493720 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-podinfo" pod "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" (UID: "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:44:30.493952 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.493927 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-kube-api-access-btz5g" pod "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" (UID: "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231"). InnerVolumeSpecName "kube-api-access-btz5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:30.494203 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.494182 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-token" pod "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" (UID: "fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:30.592552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.592458 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.592552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.592510 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.592552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.592523 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.592552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.592534 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.592552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.592544 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.592552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.592556 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.592876 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.592565 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:30.592876 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:30.592573 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btz5g\" (UniqueName: \"kubernetes.io/projected/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231-kube-api-access-btz5g\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:31.042475 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.042435 7010 generic.go:356] "Generic (PLEG): container finished" podID="fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" containerID="ce05764bdeb208538cf70152a554d5136ccb8828f2a2777a225919e99ab6725a" exitCode=0 Oct 30 13:44:31.042677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.042513 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" Oct 30 13:44:31.042677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.042530 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" event={"ID":"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231","Type":"ContainerDied","Data":"ce05764bdeb208538cf70152a554d5136ccb8828f2a2777a225919e99ab6725a"} Oct 30 13:44:31.042677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.042606 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw" event={"ID":"fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231","Type":"ContainerDied","Data":"9ae0c6af16c43df717fc211d50d68686c764a332dfd288496308e412115743cf"} Oct 30 13:44:31.042677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.042641 7010 scope.go:122] "RemoveContainer" containerID="ce05764bdeb208538cf70152a554d5136ccb8828f2a2777a225919e99ab6725a" Oct 30 13:44:31.045811 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.045774 7010 generic.go:356] "Generic (PLEG): container finished" podID="a14c1b45-7310-4620-bc17-07354849809e" containerID="27d7d6d97303afbd413eb046a4d4f8625dcfdb1df030ec554d658771d9db3db8" exitCode=0 Oct 30 13:44:31.046286 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.045830 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" Oct 30 13:44:31.046286 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.045846 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" event={"ID":"a14c1b45-7310-4620-bc17-07354849809e","Type":"ContainerDied","Data":"27d7d6d97303afbd413eb046a4d4f8625dcfdb1df030ec554d658771d9db3db8"} Oct 30 13:44:31.046286 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.045910 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh" event={"ID":"a14c1b45-7310-4620-bc17-07354849809e","Type":"ContainerDied","Data":"420aab01ff73707574f09afdcf720745a156c09d575e691e6e863ba7ab7f8012"} Oct 30 13:44:31.047861 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.047828 7010 generic.go:356] "Generic (PLEG): container finished" podID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerID="8e6091a0468e8188798e496ee01e4568f3c4077453ff8f28603063277c5a711e" exitCode=0 Oct 30 13:44:31.047994 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.047907 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" event={"ID":"b3640a4c-2e41-4186-ae55-d7a03c986793","Type":"ContainerDied","Data":"8e6091a0468e8188798e496ee01e4568f3c4077453ff8f28603063277c5a711e"} Oct 30 13:44:31.047994 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.047942 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" event={"ID":"b3640a4c-2e41-4186-ae55-d7a03c986793","Type":"ContainerDied","Data":"5026ef693f123cf78a0c058e59db3a1ef05e2390f4260abc9dc99f043bbb57f5"} Oct 30 13:44:31.048115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.048004 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq" Oct 30 13:44:31.057845 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.057822 7010 scope.go:122] "RemoveContainer" containerID="1f3037052e5d6bba927c98efbd6ee2226509efac2634621a8ad0f4c965d9bb20" Oct 30 13:44:31.090128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.090104 7010 scope.go:122] "RemoveContainer" containerID="ce05764bdeb208538cf70152a554d5136ccb8828f2a2777a225919e99ab6725a" Oct 30 13:44:31.090477 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:31.090456 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce05764bdeb208538cf70152a554d5136ccb8828f2a2777a225919e99ab6725a\": container with ID starting with ce05764bdeb208538cf70152a554d5136ccb8828f2a2777a225919e99ab6725a not found: ID does not exist" containerID="ce05764bdeb208538cf70152a554d5136ccb8828f2a2777a225919e99ab6725a" Oct 30 13:44:31.090584 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.090505 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce05764bdeb208538cf70152a554d5136ccb8828f2a2777a225919e99ab6725a"} err="failed to get container status \"ce05764bdeb208538cf70152a554d5136ccb8828f2a2777a225919e99ab6725a\": rpc error: code = NotFound desc = could not find container \"ce05764bdeb208538cf70152a554d5136ccb8828f2a2777a225919e99ab6725a\": container with ID starting with ce05764bdeb208538cf70152a554d5136ccb8828f2a2777a225919e99ab6725a not found: ID does not exist" Oct 30 13:44:31.090584 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.090526 7010 scope.go:122] "RemoveContainer" containerID="1f3037052e5d6bba927c98efbd6ee2226509efac2634621a8ad0f4c965d9bb20" Oct 30 13:44:31.090836 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:31.090811 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3037052e5d6bba927c98efbd6ee2226509efac2634621a8ad0f4c965d9bb20\": container with ID starting with 1f3037052e5d6bba927c98efbd6ee2226509efac2634621a8ad0f4c965d9bb20 not found: ID does not exist" containerID="1f3037052e5d6bba927c98efbd6ee2226509efac2634621a8ad0f4c965d9bb20" Oct 30 13:44:31.090959 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.090841 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3037052e5d6bba927c98efbd6ee2226509efac2634621a8ad0f4c965d9bb20"} err="failed to get container status \"1f3037052e5d6bba927c98efbd6ee2226509efac2634621a8ad0f4c965d9bb20\": rpc error: code = NotFound desc = could not find container \"1f3037052e5d6bba927c98efbd6ee2226509efac2634621a8ad0f4c965d9bb20\": container with ID starting with 1f3037052e5d6bba927c98efbd6ee2226509efac2634621a8ad0f4c965d9bb20 not found: ID does not exist" Oct 30 13:44:31.090959 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.090860 7010 scope.go:122] "RemoveContainer" containerID="27d7d6d97303afbd413eb046a4d4f8625dcfdb1df030ec554d658771d9db3db8" Oct 30 13:44:31.100327 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.100304 7010 scope.go:122] "RemoveContainer" containerID="27d7d6d97303afbd413eb046a4d4f8625dcfdb1df030ec554d658771d9db3db8" Oct 30 13:44:31.100714 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:31.100684 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d7d6d97303afbd413eb046a4d4f8625dcfdb1df030ec554d658771d9db3db8\": container with ID starting with 27d7d6d97303afbd413eb046a4d4f8625dcfdb1df030ec554d658771d9db3db8 not found: ID does not exist" containerID="27d7d6d97303afbd413eb046a4d4f8625dcfdb1df030ec554d658771d9db3db8" Oct 30 13:44:31.100805 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.100727 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d7d6d97303afbd413eb046a4d4f8625dcfdb1df030ec554d658771d9db3db8"} err="failed to get container status \"27d7d6d97303afbd413eb046a4d4f8625dcfdb1df030ec554d658771d9db3db8\": rpc error: code = NotFound desc = could not find container \"27d7d6d97303afbd413eb046a4d4f8625dcfdb1df030ec554d658771d9db3db8\": container with ID starting with 27d7d6d97303afbd413eb046a4d4f8625dcfdb1df030ec554d658771d9db3db8 not found: ID does not exist" Oct 30 13:44:31.100805 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.100755 7010 scope.go:122] "RemoveContainer" containerID="8e6091a0468e8188798e496ee01e4568f3c4077453ff8f28603063277c5a711e" Oct 30 13:44:31.110699 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.110676 7010 scope.go:122] "RemoveContainer" containerID="8e6091a0468e8188798e496ee01e4568f3c4077453ff8f28603063277c5a711e" Oct 30 13:44:31.111022 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:31.111000 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6091a0468e8188798e496ee01e4568f3c4077453ff8f28603063277c5a711e\": container with ID starting with 8e6091a0468e8188798e496ee01e4568f3c4077453ff8f28603063277c5a711e not found: ID does not exist" containerID="8e6091a0468e8188798e496ee01e4568f3c4077453ff8f28603063277c5a711e" Oct 30 13:44:31.111096 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.111031 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6091a0468e8188798e496ee01e4568f3c4077453ff8f28603063277c5a711e"} err="failed to get container status \"8e6091a0468e8188798e496ee01e4568f3c4077453ff8f28603063277c5a711e\": rpc error: code = NotFound desc = could not find container \"8e6091a0468e8188798e496ee01e4568f3c4077453ff8f28603063277c5a711e\": container with ID starting with 8e6091a0468e8188798e496ee01e4568f3c4077453ff8f28603063277c5a711e not found: ID does not exist" Oct 30 13:44:31.133372 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.133331 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw"] Oct 30 13:44:31.146163 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.146129 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fdd4d4-openshift-default-5bfc58469f-tk9jw"] Oct 30 13:44:31.188754 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.188724 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh"] Oct 30 13:44:31.190981 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.190955 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/distinct-w3-d833bd-kserve-6f7b95885d-njwnh"] Oct 30 13:44:31.236449 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.236413 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq"] Oct 30 13:44:31.245217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.245186 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d833bd/simulator-w3-d833bd-kserve-55f47b9fc4-kr5vq"] Oct 30 13:44:31.489252 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.489217 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a14c1b45-7310-4620-bc17-07354849809e" path="/var/lib/kubelet/pods/a14c1b45-7310-4620-bc17-07354849809e/volumes" Oct 30 13:44:31.489736 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.489720 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" path="/var/lib/kubelet/pods/b3640a4c-2e41-4186-ae55-d7a03c986793/volumes" Oct 30 13:44:31.490199 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:31.490183 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" path="/var/lib/kubelet/pods/fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231/volumes" Oct 30 13:44:33.069588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.069552 7010 generic.go:356] "Generic (PLEG): container finished" podID="5dafc909-0a2f-41dd-9529-76cbfad4ce5f" containerID="604ff51449344ccc20f224a52bc1b80c8a0919729e934939b300619299c20086" exitCode=0 Oct 30 13:44:33.070090 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.069629 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" event={"ID":"5dafc909-0a2f-41dd-9529-76cbfad4ce5f","Type":"ContainerDied","Data":"604ff51449344ccc20f224a52bc1b80c8a0919729e934939b300619299c20086"} Oct 30 13:44:33.138985 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.138954 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:44:33.227337 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.226590 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-envoy\") pod \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " Oct 30 13:44:33.227337 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.226660 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-token\" (UniqueName: \"kubernetes.io/projected/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-token\") pod \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " Oct 30 13:44:33.227337 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.226761 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istiod-ca-cert\") pod \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " Oct 30 13:44:33.227337 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.226788 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-kube-api-access-fr5m4\" (UniqueName: \"kubernetes.io/projected/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-kube-api-access-fr5m4\") pod \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " Oct 30 13:44:33.227337 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.226822 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-workload-socket\") pod \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " Oct 30 13:44:33.227337 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.226875 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-podinfo\") pod \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " Oct 30 13:44:33.227337 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.226900 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-workload-certs\") pod \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " Oct 30 13:44:33.227337 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.226940 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-data\") pod \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " Oct 30 13:44:33.227337 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.227012 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-credential-socket\") pod \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\" (UID: \"5dafc909-0a2f-41dd-9529-76cbfad4ce5f\") " Oct 30 13:44:33.229608 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.228428 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-workload-certs" pod "5dafc909-0a2f-41dd-9529-76cbfad4ce5f" (UID: "5dafc909-0a2f-41dd-9529-76cbfad4ce5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:33.229608 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.229023 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-workload-socket" pod "5dafc909-0a2f-41dd-9529-76cbfad4ce5f" (UID: "5dafc909-0a2f-41dd-9529-76cbfad4ce5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:33.229608 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.229235 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istiod-ca-cert" pod "5dafc909-0a2f-41dd-9529-76cbfad4ce5f" (UID: "5dafc909-0a2f-41dd-9529-76cbfad4ce5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:33.229999 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.229724 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-data" pod "5dafc909-0a2f-41dd-9529-76cbfad4ce5f" (UID: "5dafc909-0a2f-41dd-9529-76cbfad4ce5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:33.229999 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.229898 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-credential-socket" pod "5dafc909-0a2f-41dd-9529-76cbfad4ce5f" (UID: "5dafc909-0a2f-41dd-9529-76cbfad4ce5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:33.230412 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.230370 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-envoy" pod "5dafc909-0a2f-41dd-9529-76cbfad4ce5f" (UID: "5dafc909-0a2f-41dd-9529-76cbfad4ce5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:33.233062 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.233039 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-podinfo" pod "5dafc909-0a2f-41dd-9529-76cbfad4ce5f" (UID: "5dafc909-0a2f-41dd-9529-76cbfad4ce5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:44:33.235678 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.235638 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-token" pod "5dafc909-0a2f-41dd-9529-76cbfad4ce5f" (UID: "5dafc909-0a2f-41dd-9529-76cbfad4ce5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:33.235819 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.235798 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-kube-api-access-fr5m4" pod "5dafc909-0a2f-41dd-9529-76cbfad4ce5f" (UID: "5dafc909-0a2f-41dd-9529-76cbfad4ce5f"). InnerVolumeSpecName "kube-api-access-fr5m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:33.330468 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.330425 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.330468 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.330466 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.330768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.330479 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.330768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.330509 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr5m4\" (UniqueName: \"kubernetes.io/projected/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-kube-api-access-fr5m4\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.330768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.330525 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.330768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.330538 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.330768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.330551 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.330768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.330563 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.330768 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.330577 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dafc909-0a2f-41dd-9529-76cbfad4ce5f-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:33.967673 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.967634 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp"] Oct 30 13:44:33.967956 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:33.967898 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" podUID="1f248e80-d54a-44e8-b2b1-8ffd39b09e2f" containerName="maas-api" containerID="cri-o://5dad2fae5409780b751ebd4063cb5b5767c481aa4331b1cdebef7c816257622f" gracePeriod=30 Oct 30 13:44:34.079057 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.079026 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" Oct 30 13:44:34.079550 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.079055 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s" event={"ID":"5dafc909-0a2f-41dd-9529-76cbfad4ce5f","Type":"ContainerDied","Data":"6b3e38a83d25c927d2db8337c8c3512de4a42a90efdf0499048449df69007605"} Oct 30 13:44:34.079550 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.079133 7010 scope.go:122] "RemoveContainer" containerID="604ff51449344ccc20f224a52bc1b80c8a0919729e934939b300619299c20086" Oct 30 13:44:34.154151 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.148223 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s"] Oct 30 13:44:34.162709 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.162667 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-ff4285-openshift-default-579bcf44ff-jqt5s"] Oct 30 13:44:34.302393 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.297878 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:44:34.342703 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.342668 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-openshift-service-ca\") pod \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\" (UID: \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\") " Oct 30 13:44:34.342937 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.342811 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-kube-api-access-r996z\" (UniqueName: \"kubernetes.io/projected/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-kube-api-access-r996z\") pod \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\" (UID: \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\") " Oct 30 13:44:34.342937 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.342865 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-metrics-certs\") pod \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\" (UID: \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\") " Oct 30 13:44:34.342937 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.342903 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-maas-api-tls\") pod \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\" (UID: \"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f\") " Oct 30 13:44:34.343895 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.343864 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-openshift-service-ca" pod "1f248e80-d54a-44e8-b2b1-8ffd39b09e2f" (UID: "1f248e80-d54a-44e8-b2b1-8ffd39b09e2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:34.348670 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.348628 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-metrics-certs" pod "1f248e80-d54a-44e8-b2b1-8ffd39b09e2f" (UID: "1f248e80-d54a-44e8-b2b1-8ffd39b09e2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:34.348875 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.348714 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-maas-api-tls" pod "1f248e80-d54a-44e8-b2b1-8ffd39b09e2f" (UID: "1f248e80-d54a-44e8-b2b1-8ffd39b09e2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:44:34.349034 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.349005 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-kube-api-access-r996z" pod "1f248e80-d54a-44e8-b2b1-8ffd39b09e2f" (UID: "1f248e80-d54a-44e8-b2b1-8ffd39b09e2f"). InnerVolumeSpecName "kube-api-access-r996z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:34.445011 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.444970 7010 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-openshift-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:34.445011 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.445005 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r996z\" (UniqueName: \"kubernetes.io/projected/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-kube-api-access-r996z\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:34.445011 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.445016 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:34.445280 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:34.445024 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f-maas-api-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:35.087748 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:35.087709 7010 generic.go:356] "Generic (PLEG): container finished" podID="1f248e80-d54a-44e8-b2b1-8ffd39b09e2f" containerID="5dad2fae5409780b751ebd4063cb5b5767c481aa4331b1cdebef7c816257622f" exitCode=0 Oct 30 13:44:35.088225 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:35.087753 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" Oct 30 13:44:35.088225 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:35.087779 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" event={"ID":"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f","Type":"ContainerDied","Data":"5dad2fae5409780b751ebd4063cb5b5767c481aa4331b1cdebef7c816257622f"} Oct 30 13:44:35.088225 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:35.087827 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp" event={"ID":"1f248e80-d54a-44e8-b2b1-8ffd39b09e2f","Type":"ContainerDied","Data":"2a6e7df538b973760eb1a96083441d75e81ad6a98ec95ee4d1c876cf4d073548"} Oct 30 13:44:35.088225 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:35.087851 7010 scope.go:122] "RemoveContainer" containerID="5dad2fae5409780b751ebd4063cb5b5767c481aa4331b1cdebef7c816257622f" Oct 30 13:44:35.102696 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:35.102663 7010 scope.go:122] "RemoveContainer" containerID="5dad2fae5409780b751ebd4063cb5b5767c481aa4331b1cdebef7c816257622f" Oct 30 13:44:35.103327 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:35.103293 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dad2fae5409780b751ebd4063cb5b5767c481aa4331b1cdebef7c816257622f\": container with ID starting with 5dad2fae5409780b751ebd4063cb5b5767c481aa4331b1cdebef7c816257622f not found: ID does not exist" containerID="5dad2fae5409780b751ebd4063cb5b5767c481aa4331b1cdebef7c816257622f" Oct 30 13:44:35.103429 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:35.103340 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dad2fae5409780b751ebd4063cb5b5767c481aa4331b1cdebef7c816257622f"} err="failed to get container status \"5dad2fae5409780b751ebd4063cb5b5767c481aa4331b1cdebef7c816257622f\": rpc error: code = NotFound desc = could not find container \"5dad2fae5409780b751ebd4063cb5b5767c481aa4331b1cdebef7c816257622f\": container with ID starting with 5dad2fae5409780b751ebd4063cb5b5767c481aa4331b1cdebef7c816257622f not found: ID does not exist" Oct 30 13:44:35.142613 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:35.142563 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp"] Oct 30 13:44:35.144555 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:35.144513 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9c6033-79d5f6db55-4v7xp"] Oct 30 13:44:35.491391 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:35.491353 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f248e80-d54a-44e8-b2b1-8ffd39b09e2f" path="/var/lib/kubelet/pods/1f248e80-d54a-44e8-b2b1-8ffd39b09e2f/volumes" Oct 30 13:44:35.492146 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:35.492124 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dafc909-0a2f-41dd-9529-76cbfad4ce5f" path="/var/lib/kubelet/pods/5dafc909-0a2f-41dd-9529-76cbfad4ce5f/volumes" Oct 30 13:44:38.750377 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:38.750333 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6"] Oct 30 13:44:43.408469 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:43.408419 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c"] Oct 30 13:44:43.415997 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:43.415933 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" podUID="2d8103d2-d203-4fc6-9e85-e071908539ab" containerName="istio-proxy" containerID="cri-o://e755a2c5e593690430564e91bdd7c88542082b17f964dfb454cca982903d63c1" gracePeriod=30 Oct 30 13:44:43.751276 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:43.751179 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" podUID="246d0d04-97d3-4762-9075-777053be1de9" containerName="payload-pre-processing" containerID="cri-o://0a6b54b305cb6edd985e73c919029d7d549cec5825798bf02070a3330b0b864d" gracePeriod=25 Oct 30 13:44:44.106334 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:44.106221 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729"] Oct 30 13:44:44.106558 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:44.106515 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" podUID="3575f061-f8d0-4915-b4e8-c88c882407e5" containerName="istio-proxy" containerID="cri-o://b1a0ce1ffc50f11a1e2d7bc3aeb9352a1d93208e3f45e5e945693f044ade7dde" gracePeriod=30 Oct 30 13:44:45.525547 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.525509 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x"] Oct 30 13:44:45.526006 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.525990 7010 cpu_manager.go:414] "containerMap: removing container" podUID="ca5ec0d2-744a-4cff-a863-4aae9666eccc" containerName="payload-processing" Oct 30 13:44:45.526048 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526007 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5ec0d2-744a-4cff-a863-4aae9666eccc" containerName="payload-processing" Oct 30 13:44:45.526048 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526019 7010 cpu_manager.go:414] "containerMap: removing container" podUID="5972caf5-43d7-4006-8c85-00d82f960f84" containerName="payload-pre-processing" Oct 30 13:44:45.526048 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526025 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5972caf5-43d7-4006-8c85-00d82f960f84" containerName="payload-pre-processing" Oct 30 13:44:45.526048 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526033 7010 cpu_manager.go:414] "containerMap: removing container" podUID="84d2121e-3ecd-4540-889b-3afd02a60f91" containerName="payload-pre-processing" Oct 30 13:44:45.526048 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526038 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d2121e-3ecd-4540-889b-3afd02a60f91" containerName="payload-pre-processing" Oct 30 13:44:45.526048 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526048 7010 cpu_manager.go:414] "containerMap: removing container" podUID="a14c1b45-7310-4620-bc17-07354849809e" containerName="main" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526053 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a14c1b45-7310-4620-bc17-07354849809e" containerName="main" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526064 7010 cpu_manager.go:414] "containerMap: removing container" podUID="d8dd1343-6d33-4c14-b861-0cc98485d047" containerName="payload-pre-processing" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526069 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8dd1343-6d33-4c14-b861-0cc98485d047" containerName="payload-pre-processing" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526077 7010 cpu_manager.go:414] "containerMap: removing container" podUID="a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" containerName="istio-proxy" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526081 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" containerName="istio-proxy" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526088 7010 cpu_manager.go:414] "containerMap: removing container" podUID="0f1352f2-5503-4593-96d1-2cbbf798145e" containerName="payload-processing" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526093 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f1352f2-5503-4593-96d1-2cbbf798145e" containerName="payload-processing" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526104 7010 cpu_manager.go:414] "containerMap: removing container" podUID="ad23e4ff-651b-4e54-a180-d1cc7c73c51d" containerName="payload-processing" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526109 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad23e4ff-651b-4e54-a180-d1cc7c73c51d" containerName="payload-processing" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526117 7010 cpu_manager.go:414] "containerMap: removing container" podUID="fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" containerName="istio-proxy" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526121 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" containerName="istio-proxy" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526127 7010 cpu_manager.go:414] "containerMap: removing container" podUID="e51e863c-9c51-4df1-af43-7877a1e2c227" containerName="payload-pre-processing" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526131 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51e863c-9c51-4df1-af43-7877a1e2c227" containerName="payload-pre-processing" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526141 7010 cpu_manager.go:414] "containerMap: removing container" podUID="1f248e80-d54a-44e8-b2b1-8ffd39b09e2f" containerName="maas-api" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526145 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f248e80-d54a-44e8-b2b1-8ffd39b09e2f" containerName="maas-api" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526153 7010 cpu_manager.go:414] "containerMap: removing container" podUID="57a206b4-1557-4b43-9b49-f49f981334e9" containerName="payload-processing" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526157 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a206b4-1557-4b43-9b49-f49f981334e9" containerName="payload-processing" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526166 7010 cpu_manager.go:414] "containerMap: removing container" podUID="28dc2205-c86e-4bd3-b364-15b5a3d6aee4" containerName="payload-pre-processing" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526170 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28dc2205-c86e-4bd3-b364-15b5a3d6aee4" containerName="payload-pre-processing" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526178 7010 cpu_manager.go:414] "containerMap: removing container" podUID="7460fa64-f0e9-4542-a6cd-31d353166cc1" containerName="maas-api" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526183 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7460fa64-f0e9-4542-a6cd-31d353166cc1" containerName="maas-api" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526189 7010 cpu_manager.go:414] "containerMap: removing container" podUID="1c94ab46-fc9f-4f4d-899c-423d65bde06e" containerName="istio-proxy" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526194 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c94ab46-fc9f-4f4d-899c-423d65bde06e" containerName="istio-proxy" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526200 7010 cpu_manager.go:414] "containerMap: removing container" podUID="c5ba48c8-ab14-4261-8529-eefa097249d4" containerName="payload-pre-processing" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526206 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ba48c8-ab14-4261-8529-eefa097249d4" containerName="payload-pre-processing" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526216 7010 cpu_manager.go:414] "containerMap: removing container" podUID="4764645c-c41c-4085-8131-424e1ff46c72" containerName="istio-proxy" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526221 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4764645c-c41c-4085-8131-424e1ff46c72" containerName="istio-proxy" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526230 7010 cpu_manager.go:414] "containerMap: removing container" podUID="5dafc909-0a2f-41dd-9529-76cbfad4ce5f" containerName="istio-proxy" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526234 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dafc909-0a2f-41dd-9529-76cbfad4ce5f" containerName="istio-proxy" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526245 7010 cpu_manager.go:414] "containerMap: removing container" podUID="36b712a0-c94f-4e1d-b769-56b308d4e7b1" containerName="payload-processing" Oct 30 13:44:45.526238 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526250 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b712a0-c94f-4e1d-b769-56b308d4e7b1" containerName="payload-processing" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526257 7010 cpu_manager.go:414] "containerMap: removing container" podUID="415237a2-07c0-481d-bb7b-f818c83e92ce" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526261 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415237a2-07c0-481d-bb7b-f818c83e92ce" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526266 7010 cpu_manager.go:414] "containerMap: removing container" podUID="37d669cd-c104-458a-85c2-65d8f7e0c50a" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526271 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d669cd-c104-458a-85c2-65d8f7e0c50a" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526278 7010 cpu_manager.go:414] "containerMap: removing container" podUID="e8430e34-fc9c-42de-82d5-b4d2ffd07dfc" containerName="maas-api" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526282 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8430e34-fc9c-42de-82d5-b4d2ffd07dfc" containerName="maas-api" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526288 7010 cpu_manager.go:414] "containerMap: removing container" podUID="9e028b61-6039-4f07-b7a9-082d7b51c633" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526292 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e028b61-6039-4f07-b7a9-082d7b51c633" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526301 7010 cpu_manager.go:414] "containerMap: removing container" podUID="aa83aaa6-483b-4ae1-aaaa-2f30faf7f576" containerName="payload-processing" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526305 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa83aaa6-483b-4ae1-aaaa-2f30faf7f576" containerName="payload-processing" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526314 7010 cpu_manager.go:414] "containerMap: removing container" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526318 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526326 7010 cpu_manager.go:414] "containerMap: removing container" podUID="5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526330 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526339 7010 cpu_manager.go:414] "containerMap: removing container" podUID="415237a2-07c0-481d-bb7b-f818c83e92ce" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526343 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415237a2-07c0-481d-bb7b-f818c83e92ce" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526349 7010 cpu_manager.go:414] "containerMap: removing container" podUID="998f2069-f1df-403c-9719-46a9a3cda3fa" containerName="maas-api" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526354 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998f2069-f1df-403c-9719-46a9a3cda3fa" containerName="maas-api" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526417 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dafc909-0a2f-41dd-9529-76cbfad4ce5f" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526426 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b712a0-c94f-4e1d-b769-56b308d4e7b1" containerName="payload-processing" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526431 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="415237a2-07c0-481d-bb7b-f818c83e92ce" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526436 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d669cd-c104-458a-85c2-65d8f7e0c50a" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526441 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8430e34-fc9c-42de-82d5-b4d2ffd07dfc" containerName="maas-api" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526446 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e028b61-6039-4f07-b7a9-082d7b51c633" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526453 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa83aaa6-483b-4ae1-aaaa-2f30faf7f576" containerName="payload-processing" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526460 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3640a4c-2e41-4186-ae55-d7a03c986793" containerName="main" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526466 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f15b5e8-0877-4f50-ae90-6bfaab9a5fb5" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526471 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="415237a2-07c0-481d-bb7b-f818c83e92ce" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526476 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="998f2069-f1df-403c-9719-46a9a3cda3fa" containerName="maas-api" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526483 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5ec0d2-744a-4cff-a863-4aae9666eccc" containerName="payload-processing" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526503 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="5972caf5-43d7-4006-8c85-00d82f960f84" containerName="payload-pre-processing" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526510 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d2121e-3ecd-4540-889b-3afd02a60f91" containerName="payload-pre-processing" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526519 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="a14c1b45-7310-4620-bc17-07354849809e" containerName="main" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526530 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8dd1343-6d33-4c14-b861-0cc98485d047" containerName="payload-pre-processing" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526537 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526545 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f1352f2-5503-4593-96d1-2cbbf798145e" containerName="payload-processing" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526554 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad23e4ff-651b-4e54-a180-d1cc7c73c51d" containerName="payload-processing" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526564 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526569 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51e863c-9c51-4df1-af43-7877a1e2c227" containerName="payload-pre-processing" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526575 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f248e80-d54a-44e8-b2b1-8ffd39b09e2f" containerName="maas-api" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526582 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526587 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a206b4-1557-4b43-9b49-f49f981334e9" containerName="payload-processing" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526594 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="28dc2205-c86e-4bd3-b364-15b5a3d6aee4" containerName="payload-pre-processing" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526599 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="7460fa64-f0e9-4542-a6cd-31d353166cc1" containerName="maas-api" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526605 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c94ab46-fc9f-4f4d-899c-423d65bde06e" containerName="istio-proxy" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526611 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ba48c8-ab14-4261-8529-eefa097249d4" containerName="payload-pre-processing" Oct 30 13:44:45.527221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.526617 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="4764645c-c41c-4085-8131-424e1ff46c72" containerName="istio-proxy" Oct 30 13:44:45.531874 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.531854 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.553550 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.553514 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x"] Oct 30 13:44:45.568036 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.568001 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d833bd-openshift-default-dockercfg-dpcql\"" Oct 30 13:44:45.589940 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.589902 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.589940 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.589946 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.590233 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.590048 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.590233 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.590118 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.590233 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.590154 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.590370 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.590244 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.590370 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.590319 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.590441 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.590375 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.590521 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.590500 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9g65\" (UniqueName: \"kubernetes.io/projected/78b88187-8364-42ca-84dd-9e3d06c1ee3b-kube-api-access-z9g65\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.691993 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.691956 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.691993 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.692009 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.692277 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.692054 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9g65\" (UniqueName: \"kubernetes.io/projected/78b88187-8364-42ca-84dd-9e3d06c1ee3b-kube-api-access-z9g65\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.692277 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.692089 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.692277 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.692105 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.692277 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.692140 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.692277 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.692190 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.692277 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.692225 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.692277 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.692265 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.692706 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.692679 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-workload-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.692784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.692758 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-workload-certs\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.692784 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.692771 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-data\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.692905 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.692807 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-credential-socket\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.692994 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.692974 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istiod-ca-cert\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.694541 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.694517 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-envoy\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.694797 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.694779 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-podinfo\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.717769 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.717735 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9g65\" (UniqueName: \"kubernetes.io/projected/78b88187-8364-42ca-84dd-9e3d06c1ee3b-kube-api-access-z9g65\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.718085 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.718058 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-token\") pod \"e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:45.846974 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:45.846881 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:46.223510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:46.206081 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x"] Oct 30 13:44:46.223510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:46.218508 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:44:46.223510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:46.218620 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:44:46.223510 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:46.218661 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:44:47.173336 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:47.173294 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" event={"ID":"78b88187-8364-42ca-84dd-9e3d06c1ee3b","Type":"ContainerStarted","Data":"d6451422ee5b90a5a8b801460ce95e3d8f45936d165e010a7bb3ae06ada2a724"} Oct 30 13:44:47.173996 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:47.173366 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" event={"ID":"78b88187-8364-42ca-84dd-9e3d06c1ee3b","Type":"ContainerStarted","Data":"93ba4decb137977f943588f4595d32a04ab5ce233ec6643b7b33e62b94d78766"} Oct 30 13:44:47.258127 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:47.258052 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" podStartSLOduration=2.258027393 podStartE2EDuration="2.258027393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:47.24240279 +0000 UTC m=+3143.874589946" watchObservedRunningTime="2026-10-30 13:44:47.258027393 +0000 UTC m=+3143.890214556" Oct 30 13:44:47.847580 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:47.847514 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:47.849196 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:47.849168 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 13:44:47.849327 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:47.849218 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" podUID="78b88187-8364-42ca-84dd-9e3d06c1ee3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 13:44:48.133778 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.133741 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" podUID="246d0d04-97d3-4762-9075-777053be1de9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:44:48.172821 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.172785 7010 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:44:48.172982 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.172845 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" podUID="3575f061-f8d0-4915-b4e8-c88c882407e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:44:48.740338 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.740226 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:44:48.744252 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.744218 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8103d2-d203-4fc6-9e85-e071908539ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8103d2-d203-4fc6-9e85-e071908539ab-istiod-ca-cert\") pod \"2d8103d2-d203-4fc6-9e85-e071908539ab\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " Oct 30 13:44:48.744400 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.744327 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-workload-certs\") pod \"2d8103d2-d203-4fc6-9e85-e071908539ab\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " Oct 30 13:44:48.744400 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.744369 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-token\" (UniqueName: \"kubernetes.io/projected/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-token\") pod \"2d8103d2-d203-4fc6-9e85-e071908539ab\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " Oct 30 13:44:48.744571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.744445 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-envoy\") pod \"2d8103d2-d203-4fc6-9e85-e071908539ab\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " Oct 30 13:44:48.744571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.744528 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-podinfo\") pod \"2d8103d2-d203-4fc6-9e85-e071908539ab\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " Oct 30 13:44:48.744695 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.744596 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8103d2-d203-4fc6-9e85-e071908539ab-kube-api-access-nktmp\" (UniqueName: \"kubernetes.io/projected/2d8103d2-d203-4fc6-9e85-e071908539ab-kube-api-access-nktmp\") pod \"2d8103d2-d203-4fc6-9e85-e071908539ab\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " Oct 30 13:44:48.744695 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.744627 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-credential-socket\") pod \"2d8103d2-d203-4fc6-9e85-e071908539ab\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " Oct 30 13:44:48.744794 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.744701 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-data\") pod \"2d8103d2-d203-4fc6-9e85-e071908539ab\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " Oct 30 13:44:48.744794 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.744737 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-workload-socket\") pod \"2d8103d2-d203-4fc6-9e85-e071908539ab\" (UID: \"2d8103d2-d203-4fc6-9e85-e071908539ab\") " Oct 30 13:44:48.744794 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.744736 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8103d2-d203-4fc6-9e85-e071908539ab-istiod-ca-cert" pod "2d8103d2-d203-4fc6-9e85-e071908539ab" (UID: "2d8103d2-d203-4fc6-9e85-e071908539ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:48.745011 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.744984 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-workload-certs" pod "2d8103d2-d203-4fc6-9e85-e071908539ab" (UID: "2d8103d2-d203-4fc6-9e85-e071908539ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:48.745076 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.745046 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-credential-socket" pod "2d8103d2-d203-4fc6-9e85-e071908539ab" (UID: "2d8103d2-d203-4fc6-9e85-e071908539ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:48.745138 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.745063 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-data" pod "2d8103d2-d203-4fc6-9e85-e071908539ab" (UID: "2d8103d2-d203-4fc6-9e85-e071908539ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:48.745227 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.745208 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-workload-socket" pod "2d8103d2-d203-4fc6-9e85-e071908539ab" (UID: "2d8103d2-d203-4fc6-9e85-e071908539ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:48.745303 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.745251 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8103d2-d203-4fc6-9e85-e071908539ab-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:48.745303 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.745265 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:48.745303 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.745274 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:48.745303 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.745283 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:48.747203 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.747118 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-envoy" pod "2d8103d2-d203-4fc6-9e85-e071908539ab" (UID: "2d8103d2-d203-4fc6-9e85-e071908539ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:48.747437 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.747398 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-podinfo" pod "2d8103d2-d203-4fc6-9e85-e071908539ab" (UID: "2d8103d2-d203-4fc6-9e85-e071908539ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:44:48.747842 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.747813 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-token" pod "2d8103d2-d203-4fc6-9e85-e071908539ab" (UID: "2d8103d2-d203-4fc6-9e85-e071908539ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:48.748085 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.748053 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8103d2-d203-4fc6-9e85-e071908539ab-kube-api-access-nktmp" pod "2d8103d2-d203-4fc6-9e85-e071908539ab" (UID: "2d8103d2-d203-4fc6-9e85-e071908539ab"). InnerVolumeSpecName "kube-api-access-nktmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:48.850168 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.850123 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 13:44:48.850394 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.850183 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" podUID="78b88187-8364-42ca-84dd-9e3d06c1ee3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 13:44:48.850394 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.850205 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:48.850394 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.850239 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:48.850394 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.850254 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:48.850394 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.850267 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8103d2-d203-4fc6-9e85-e071908539ab-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:48.850394 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.850281 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nktmp\" (UniqueName: \"kubernetes.io/projected/2d8103d2-d203-4fc6-9e85-e071908539ab-kube-api-access-nktmp\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:48.927248 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.927202 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk"] Oct 30 13:44:48.936874 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:48.936836 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4"] Oct 30 13:44:49.201511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.198347 7010 generic.go:356] "Generic (PLEG): container finished" podID="2d8103d2-d203-4fc6-9e85-e071908539ab" containerID="e755a2c5e593690430564e91bdd7c88542082b17f964dfb454cca982903d63c1" exitCode=0 Oct 30 13:44:49.201511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.199390 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" Oct 30 13:44:49.208552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.202054 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" event={"ID":"2d8103d2-d203-4fc6-9e85-e071908539ab","Type":"ContainerDied","Data":"e755a2c5e593690430564e91bdd7c88542082b17f964dfb454cca982903d63c1"} Oct 30 13:44:49.208552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.202131 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c" event={"ID":"2d8103d2-d203-4fc6-9e85-e071908539ab","Type":"ContainerDied","Data":"8514bc47a950251035678382ef40163c365fb16d77285ee0bdd5a020a474aa7d"} Oct 30 13:44:49.208552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.202170 7010 scope.go:122] "RemoveContainer" containerID="e755a2c5e593690430564e91bdd7c88542082b17f964dfb454cca982903d63c1" Oct 30 13:44:49.222303 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.222268 7010 scope.go:122] "RemoveContainer" containerID="e755a2c5e593690430564e91bdd7c88542082b17f964dfb454cca982903d63c1" Oct 30 13:44:49.222703 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:49.222628 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e755a2c5e593690430564e91bdd7c88542082b17f964dfb454cca982903d63c1\": container with ID starting with e755a2c5e593690430564e91bdd7c88542082b17f964dfb454cca982903d63c1 not found: ID does not exist" containerID="e755a2c5e593690430564e91bdd7c88542082b17f964dfb454cca982903d63c1" Oct 30 13:44:49.222703 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.222657 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e755a2c5e593690430564e91bdd7c88542082b17f964dfb454cca982903d63c1"} err="failed to get container status \"e755a2c5e593690430564e91bdd7c88542082b17f964dfb454cca982903d63c1\": rpc error: code = NotFound desc = could not find container \"e755a2c5e593690430564e91bdd7c88542082b17f964dfb454cca982903d63c1\": container with ID starting with e755a2c5e593690430564e91bdd7c88542082b17f964dfb454cca982903d63c1 not found: ID does not exist" Oct 30 13:44:49.271343 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.271306 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c"] Oct 30 13:44:49.274428 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.274389 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-82796d-openshift-default-75fcc954cf-k4d7c"] Oct 30 13:44:49.453143 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.452864 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz"] Oct 30 13:44:49.453251 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.453157 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" podUID="7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" containerName="istio-proxy" containerID="cri-o://a2f8af7ed34db1bc7f9d281b72d87aeeaf4efe29d638fe85b6b1b770efa30790" gracePeriod=30 Oct 30 13:44:49.481698 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.481663 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:44:49.541747 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.540656 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8103d2-d203-4fc6-9e85-e071908539ab" path="/var/lib/kubelet/pods/2d8103d2-d203-4fc6-9e85-e071908539ab/volumes" Oct 30 13:44:49.547520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.544233 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb"] Oct 30 13:44:49.547520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.544481 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" podUID="36015964-d36a-428a-8c47-367689174d12" containerName="istio-proxy" containerID="cri-o://a03998313cc8485ece3331d041d6bd2a13aa7913114d68ec83ec815d894652bc" gracePeriod=30 Oct 30 13:44:49.562743 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.562710 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-token\" (UniqueName: \"kubernetes.io/projected/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-token\") pod \"3575f061-f8d0-4915-b4e8-c88c882407e5\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " Oct 30 13:44:49.562909 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.562775 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3575f061-f8d0-4915-b4e8-c88c882407e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3575f061-f8d0-4915-b4e8-c88c882407e5-istiod-ca-cert\") pod \"3575f061-f8d0-4915-b4e8-c88c882407e5\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " Oct 30 13:44:49.562909 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.562835 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-credential-socket\") pod \"3575f061-f8d0-4915-b4e8-c88c882407e5\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " Oct 30 13:44:49.563041 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.562911 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-workload-socket\") pod \"3575f061-f8d0-4915-b4e8-c88c882407e5\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " Oct 30 13:44:49.563105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.563035 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-envoy\") pod \"3575f061-f8d0-4915-b4e8-c88c882407e5\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " Oct 30 13:44:49.563105 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.563065 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3575f061-f8d0-4915-b4e8-c88c882407e5-kube-api-access-zrxkp\" (UniqueName: \"kubernetes.io/projected/3575f061-f8d0-4915-b4e8-c88c882407e5-kube-api-access-zrxkp\") pod \"3575f061-f8d0-4915-b4e8-c88c882407e5\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " Oct 30 13:44:49.563221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.563195 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-data\") pod \"3575f061-f8d0-4915-b4e8-c88c882407e5\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " Oct 30 13:44:49.563274 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.563247 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-podinfo\") pod \"3575f061-f8d0-4915-b4e8-c88c882407e5\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " Oct 30 13:44:49.563332 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.563300 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-workload-certs\") pod \"3575f061-f8d0-4915-b4e8-c88c882407e5\" (UID: \"3575f061-f8d0-4915-b4e8-c88c882407e5\") " Oct 30 13:44:49.563399 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.563355 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-credential-socket" pod "3575f061-f8d0-4915-b4e8-c88c882407e5" (UID: "3575f061-f8d0-4915-b4e8-c88c882407e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:49.563838 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.563661 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3575f061-f8d0-4915-b4e8-c88c882407e5-istiod-ca-cert" pod "3575f061-f8d0-4915-b4e8-c88c882407e5" (UID: "3575f061-f8d0-4915-b4e8-c88c882407e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:49.563968 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.563894 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-workload-socket" pod "3575f061-f8d0-4915-b4e8-c88c882407e5" (UID: "3575f061-f8d0-4915-b4e8-c88c882407e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:49.564053 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.564024 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:49.564111 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.564069 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:49.564111 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.564086 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3575f061-f8d0-4915-b4e8-c88c882407e5-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:49.564220 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.564116 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-data" pod "3575f061-f8d0-4915-b4e8-c88c882407e5" (UID: "3575f061-f8d0-4915-b4e8-c88c882407e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:49.564584 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.564558 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-workload-certs" pod "3575f061-f8d0-4915-b4e8-c88c882407e5" (UID: "3575f061-f8d0-4915-b4e8-c88c882407e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:49.578275 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.578225 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-envoy" pod "3575f061-f8d0-4915-b4e8-c88c882407e5" (UID: "3575f061-f8d0-4915-b4e8-c88c882407e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:49.580113 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.580072 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-token" pod "3575f061-f8d0-4915-b4e8-c88c882407e5" (UID: "3575f061-f8d0-4915-b4e8-c88c882407e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:49.580732 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.580693 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-podinfo" pod "3575f061-f8d0-4915-b4e8-c88c882407e5" (UID: "3575f061-f8d0-4915-b4e8-c88c882407e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:44:49.581896 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.581869 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3575f061-f8d0-4915-b4e8-c88c882407e5-kube-api-access-zrxkp" pod "3575f061-f8d0-4915-b4e8-c88c882407e5" (UID: "3575f061-f8d0-4915-b4e8-c88c882407e5"). InnerVolumeSpecName "kube-api-access-zrxkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:49.665100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.665052 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:49.665100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.665089 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:49.665100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.665098 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrxkp\" (UniqueName: \"kubernetes.io/projected/3575f061-f8d0-4915-b4e8-c88c882407e5-kube-api-access-zrxkp\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:49.665100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.665106 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:49.665100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.665115 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3575f061-f8d0-4915-b4e8-c88c882407e5-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:49.665573 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.665123 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3575f061-f8d0-4915-b4e8-c88c882407e5-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:49.848172 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.848087 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 13:44:49.848172 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:49.848140 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" podUID="78b88187-8364-42ca-84dd-9e3d06c1ee3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 13:44:50.215709 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:50.215648 7010 generic.go:356] "Generic (PLEG): container finished" podID="3575f061-f8d0-4915-b4e8-c88c882407e5" containerID="b1a0ce1ffc50f11a1e2d7bc3aeb9352a1d93208e3f45e5e945693f044ade7dde" exitCode=0 Oct 30 13:44:50.215939 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:50.215717 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" Oct 30 13:44:50.215939 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:50.215762 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" event={"ID":"3575f061-f8d0-4915-b4e8-c88c882407e5","Type":"ContainerDied","Data":"b1a0ce1ffc50f11a1e2d7bc3aeb9352a1d93208e3f45e5e945693f044ade7dde"} Oct 30 13:44:50.215939 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:50.215841 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729" event={"ID":"3575f061-f8d0-4915-b4e8-c88c882407e5","Type":"ContainerDied","Data":"86da82f99c287fe240536c3535e410db85c5359ff1a01897c5bc3dd996dc5eae"} Oct 30 13:44:50.215939 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:50.215867 7010 scope.go:122] "RemoveContainer" containerID="b1a0ce1ffc50f11a1e2d7bc3aeb9352a1d93208e3f45e5e945693f044ade7dde" Oct 30 13:44:50.226981 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:50.226951 7010 scope.go:122] "RemoveContainer" containerID="d4629e7d65510a170b67d2a8e9336306456a592326d8dd6387f9ee30a3a4a799" Oct 30 13:44:50.237104 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:50.237081 7010 scope.go:122] "RemoveContainer" containerID="b1a0ce1ffc50f11a1e2d7bc3aeb9352a1d93208e3f45e5e945693f044ade7dde" Oct 30 13:44:50.237454 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:50.237425 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a0ce1ffc50f11a1e2d7bc3aeb9352a1d93208e3f45e5e945693f044ade7dde\": container with ID starting with b1a0ce1ffc50f11a1e2d7bc3aeb9352a1d93208e3f45e5e945693f044ade7dde not found: ID does not exist" containerID="b1a0ce1ffc50f11a1e2d7bc3aeb9352a1d93208e3f45e5e945693f044ade7dde" Oct 30 13:44:50.237677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:50.237464 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a0ce1ffc50f11a1e2d7bc3aeb9352a1d93208e3f45e5e945693f044ade7dde"} err="failed to get container status \"b1a0ce1ffc50f11a1e2d7bc3aeb9352a1d93208e3f45e5e945693f044ade7dde\": rpc error: code = NotFound desc = could not find container \"b1a0ce1ffc50f11a1e2d7bc3aeb9352a1d93208e3f45e5e945693f044ade7dde\": container with ID starting with b1a0ce1ffc50f11a1e2d7bc3aeb9352a1d93208e3f45e5e945693f044ade7dde not found: ID does not exist" Oct 30 13:44:50.237677 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:50.237483 7010 scope.go:122] "RemoveContainer" containerID="d4629e7d65510a170b67d2a8e9336306456a592326d8dd6387f9ee30a3a4a799" Oct 30 13:44:50.237827 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:50.237788 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4629e7d65510a170b67d2a8e9336306456a592326d8dd6387f9ee30a3a4a799\": container with ID starting with d4629e7d65510a170b67d2a8e9336306456a592326d8dd6387f9ee30a3a4a799 not found: ID does not exist" containerID="d4629e7d65510a170b67d2a8e9336306456a592326d8dd6387f9ee30a3a4a799" Oct 30 13:44:50.237876 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:50.237826 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4629e7d65510a170b67d2a8e9336306456a592326d8dd6387f9ee30a3a4a799"} err="failed to get container status \"d4629e7d65510a170b67d2a8e9336306456a592326d8dd6387f9ee30a3a4a799\": rpc error: code = NotFound desc = could not find container \"d4629e7d65510a170b67d2a8e9336306456a592326d8dd6387f9ee30a3a4a799\": container with ID starting with d4629e7d65510a170b67d2a8e9336306456a592326d8dd6387f9ee30a3a4a799 not found: ID does not exist" Oct 30 13:44:50.273253 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:50.273215 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729"] Oct 30 13:44:50.280195 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:50.280162 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9c6033-openshift-default-7b475fd8bc-j8729"] Oct 30 13:44:50.847311 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:50.847278 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 13:44:50.847520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:50.847342 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" podUID="78b88187-8364-42ca-84dd-9e3d06c1ee3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 13:44:51.505515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:51.499338 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3575f061-f8d0-4915-b4e8-c88c882407e5" path="/var/lib/kubelet/pods/3575f061-f8d0-4915-b4e8-c88c882407e5/volumes" Oct 30 13:44:51.847464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:51.847371 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 13:44:51.847464 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:51.847434 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" podUID="78b88187-8364-42ca-84dd-9e3d06c1ee3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 13:44:52.847967 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:52.847927 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 13:44:52.848405 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:52.847987 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" podUID="78b88187-8364-42ca-84dd-9e3d06c1ee3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 13:44:53.848328 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:53.848287 7010 patch_prober.go:29] interesting pod/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 13:44:53.848780 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:53.848353 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" podUID="78b88187-8364-42ca-84dd-9e3d06c1ee3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 13:44:54.360054 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.360005 7010 patch_prober.go:29] interesting pod/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:44:54.360308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.360081 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" podUID="36015964-d36a-428a-8c47-367689174d12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:44:54.765595 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.763009 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:44:54.881995 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.881962 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-data\") pod \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " Oct 30 13:44:54.884021 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.883982 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-workload-certs\") pod \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " Oct 30 13:44:54.884160 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.884129 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-podinfo\") pod \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " Oct 30 13:44:54.884308 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.884293 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-token\" (UniqueName: \"kubernetes.io/projected/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-token\") pod \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " Oct 30 13:44:54.884552 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.884480 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-envoy\") pod \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " Oct 30 13:44:54.884847 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.884832 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-workload-socket\") pod \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " Oct 30 13:44:54.884986 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.884971 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-kube-api-access-7l7g2\" (UniqueName: \"kubernetes.io/projected/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-kube-api-access-7l7g2\") pod \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " Oct 30 13:44:54.885115 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.885103 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-credential-socket\") pod \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " Oct 30 13:44:54.885350 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.885334 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istiod-ca-cert\") pod \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\" (UID: \"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e\") " Oct 30 13:44:54.888103 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.886766 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-workload-certs" pod "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" (UID: "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:54.888103 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.886836 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-credential-socket" pod "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" (UID: "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:54.888103 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.886879 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-workload-socket" pod "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" (UID: "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:54.888620 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.888591 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-data" pod "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" (UID: "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:54.891227 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.891197 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istiod-ca-cert" pod "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" (UID: "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:54.891681 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.891659 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-envoy" pod "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" (UID: "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:54.891847 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.891727 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-token" pod "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" (UID: "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:54.897048 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.897007 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-kube-api-access-7l7g2" pod "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" (UID: "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e"). InnerVolumeSpecName "kube-api-access-7l7g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:54.899646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.899619 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-podinfo" pod "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" (UID: "7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:44:54.904762 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.904733 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:44:54.965035 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.964996 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn"] Oct 30 13:44:54.988118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.988074 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:54.988118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.988110 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l7g2\" (UniqueName: \"kubernetes.io/projected/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-kube-api-access-7l7g2\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:54.988118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.988120 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:54.988118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.988129 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:54.988529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.988138 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:54.988529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.988147 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:54.988529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.988155 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:54.988529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.988166 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:54.988529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:54.988180 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.001753 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.001703 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq"] Oct 30 13:44:55.089353 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.089315 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-workload-socket\") pod \"36015964-d36a-428a-8c47-367689174d12\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " Oct 30 13:44:55.089559 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.089415 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-credential-socket\") pod \"36015964-d36a-428a-8c47-367689174d12\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " Oct 30 13:44:55.089559 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.089457 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-istio-data\") pod \"36015964-d36a-428a-8c47-367689174d12\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " Oct 30 13:44:55.089559 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.089546 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36015964-d36a-428a-8c47-367689174d12-istio-token\" (UniqueName: \"kubernetes.io/projected/36015964-d36a-428a-8c47-367689174d12-istio-token\") pod \"36015964-d36a-428a-8c47-367689174d12\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " Oct 30 13:44:55.089754 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.089597 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-workload-certs\") pod \"36015964-d36a-428a-8c47-367689174d12\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " Oct 30 13:44:55.089754 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.089689 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36015964-d36a-428a-8c47-367689174d12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36015964-d36a-428a-8c47-367689174d12-istiod-ca-cert\") pod \"36015964-d36a-428a-8c47-367689174d12\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " Oct 30 13:44:55.089754 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.089720 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-istio-envoy\") pod \"36015964-d36a-428a-8c47-367689174d12\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " Oct 30 13:44:55.089754 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.089740 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36015964-d36a-428a-8c47-367689174d12-kube-api-access-9zmnr\" (UniqueName: \"kubernetes.io/projected/36015964-d36a-428a-8c47-367689174d12-kube-api-access-9zmnr\") pod \"36015964-d36a-428a-8c47-367689174d12\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " Oct 30 13:44:55.089958 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.089772 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36015964-d36a-428a-8c47-367689174d12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36015964-d36a-428a-8c47-367689174d12-istio-podinfo\") pod \"36015964-d36a-428a-8c47-367689174d12\" (UID: \"36015964-d36a-428a-8c47-367689174d12\") " Oct 30 13:44:55.090075 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.090052 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-workload-socket" pod "36015964-d36a-428a-8c47-367689174d12" (UID: "36015964-d36a-428a-8c47-367689174d12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:55.090145 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.090081 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-credential-socket" pod "36015964-d36a-428a-8c47-367689174d12" (UID: "36015964-d36a-428a-8c47-367689174d12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:55.090301 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.090285 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.090385 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.090303 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.090385 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.090337 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-workload-certs" pod "36015964-d36a-428a-8c47-367689174d12" (UID: "36015964-d36a-428a-8c47-367689174d12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:55.091005 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.090977 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36015964-d36a-428a-8c47-367689174d12-istiod-ca-cert" pod "36015964-d36a-428a-8c47-367689174d12" (UID: "36015964-d36a-428a-8c47-367689174d12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:44:55.091133 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.091095 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-istio-data" pod "36015964-d36a-428a-8c47-367689174d12" (UID: "36015964-d36a-428a-8c47-367689174d12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:55.092851 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.092825 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36015964-d36a-428a-8c47-367689174d12-istio-podinfo" pod "36015964-d36a-428a-8c47-367689174d12" (UID: "36015964-d36a-428a-8c47-367689174d12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:44:55.092851 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.092839 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-istio-envoy" pod "36015964-d36a-428a-8c47-367689174d12" (UID: "36015964-d36a-428a-8c47-367689174d12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:44:55.093296 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.093260 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36015964-d36a-428a-8c47-367689174d12-istio-token" pod "36015964-d36a-428a-8c47-367689174d12" (UID: "36015964-d36a-428a-8c47-367689174d12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:55.093296 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.093284 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36015964-d36a-428a-8c47-367689174d12-kube-api-access-9zmnr" pod "36015964-d36a-428a-8c47-367689174d12" (UID: "36015964-d36a-428a-8c47-367689174d12"). InnerVolumeSpecName "kube-api-access-9zmnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:44:55.124920 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.124889 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:55.125221 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.125199 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:55.126109 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.126089 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:44:55.191665 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.191563 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36015964-d36a-428a-8c47-367689174d12-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.191665 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.191597 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.191665 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.191610 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zmnr\" (UniqueName: \"kubernetes.io/projected/36015964-d36a-428a-8c47-367689174d12-kube-api-access-9zmnr\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.191665 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.191625 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36015964-d36a-428a-8c47-367689174d12-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.191665 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.191639 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.191665 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.191650 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36015964-d36a-428a-8c47-367689174d12-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.191665 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.191662 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36015964-d36a-428a-8c47-367689174d12-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:44:55.261395 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.261360 7010 generic.go:356] "Generic (PLEG): container finished" podID="36015964-d36a-428a-8c47-367689174d12" containerID="a03998313cc8485ece3331d041d6bd2a13aa7913114d68ec83ec815d894652bc" exitCode=0 Oct 30 13:44:55.261628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.261404 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" Oct 30 13:44:55.261628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.261456 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" event={"ID":"36015964-d36a-428a-8c47-367689174d12","Type":"ContainerDied","Data":"a03998313cc8485ece3331d041d6bd2a13aa7913114d68ec83ec815d894652bc"} Oct 30 13:44:55.261628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.261538 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb" event={"ID":"36015964-d36a-428a-8c47-367689174d12","Type":"ContainerDied","Data":"ef02862102e1d880183748b2442e14d4a51508d27c2185f11a35f4c71c79648f"} Oct 30 13:44:55.261628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.261557 7010 scope.go:122] "RemoveContainer" containerID="a03998313cc8485ece3331d041d6bd2a13aa7913114d68ec83ec815d894652bc" Oct 30 13:44:55.263095 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.263070 7010 generic.go:356] "Generic (PLEG): container finished" podID="7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" containerID="a2f8af7ed34db1bc7f9d281b72d87aeeaf4efe29d638fe85b6b1b770efa30790" exitCode=0 Oct 30 13:44:55.263209 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.263103 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" event={"ID":"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e","Type":"ContainerDied","Data":"a2f8af7ed34db1bc7f9d281b72d87aeeaf4efe29d638fe85b6b1b770efa30790"} Oct 30 13:44:55.263209 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.263170 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" event={"ID":"7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e","Type":"ContainerDied","Data":"0a4dd2b24ec73684a9742f3121ca4a22ae0e3401b88dfd1c7a1564f0a3f0deda"} Oct 30 13:44:55.263294 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.263214 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz" Oct 30 13:44:55.272574 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.272554 7010 scope.go:122] "RemoveContainer" containerID="a03998313cc8485ece3331d041d6bd2a13aa7913114d68ec83ec815d894652bc" Oct 30 13:44:55.272851 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:55.272833 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a03998313cc8485ece3331d041d6bd2a13aa7913114d68ec83ec815d894652bc\": container with ID starting with a03998313cc8485ece3331d041d6bd2a13aa7913114d68ec83ec815d894652bc not found: ID does not exist" containerID="a03998313cc8485ece3331d041d6bd2a13aa7913114d68ec83ec815d894652bc" Oct 30 13:44:55.272902 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.272860 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a03998313cc8485ece3331d041d6bd2a13aa7913114d68ec83ec815d894652bc"} err="failed to get container status \"a03998313cc8485ece3331d041d6bd2a13aa7913114d68ec83ec815d894652bc\": rpc error: code = NotFound desc = could not find container \"a03998313cc8485ece3331d041d6bd2a13aa7913114d68ec83ec815d894652bc\": container with ID starting with a03998313cc8485ece3331d041d6bd2a13aa7913114d68ec83ec815d894652bc not found: ID does not exist" Oct 30 13:44:55.272902 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.272876 7010 scope.go:122] "RemoveContainer" containerID="a2f8af7ed34db1bc7f9d281b72d87aeeaf4efe29d638fe85b6b1b770efa30790" Oct 30 13:44:55.283862 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.283841 7010 scope.go:122] "RemoveContainer" containerID="a2f8af7ed34db1bc7f9d281b72d87aeeaf4efe29d638fe85b6b1b770efa30790" Oct 30 13:44:55.284127 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:55.284110 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f8af7ed34db1bc7f9d281b72d87aeeaf4efe29d638fe85b6b1b770efa30790\": container with ID starting with a2f8af7ed34db1bc7f9d281b72d87aeeaf4efe29d638fe85b6b1b770efa30790 not found: ID does not exist" containerID="a2f8af7ed34db1bc7f9d281b72d87aeeaf4efe29d638fe85b6b1b770efa30790" Oct 30 13:44:55.284169 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.284137 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f8af7ed34db1bc7f9d281b72d87aeeaf4efe29d638fe85b6b1b770efa30790"} err="failed to get container status \"a2f8af7ed34db1bc7f9d281b72d87aeeaf4efe29d638fe85b6b1b770efa30790\": rpc error: code = NotFound desc = could not find container \"a2f8af7ed34db1bc7f9d281b72d87aeeaf4efe29d638fe85b6b1b770efa30790\": container with ID starting with a2f8af7ed34db1bc7f9d281b72d87aeeaf4efe29d638fe85b6b1b770efa30790 not found: ID does not exist" Oct 30 13:44:55.352697 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.352667 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz"] Oct 30 13:44:55.354348 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.354322 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ndmmz"] Oct 30 13:44:55.408543 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.408501 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb"] Oct 30 13:44:55.413931 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.413878 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-8d82d5-openshift-default-57dcf6bdfc-ggmpb"] Oct 30 13:44:55.502821 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.502762 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36015964-d36a-428a-8c47-367689174d12" path="/var/lib/kubelet/pods/36015964-d36a-428a-8c47-367689174d12/volumes" Oct 30 13:44:55.504053 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:55.504023 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" path="/var/lib/kubelet/pods/7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e/volumes" Oct 30 13:44:57.754178 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:57.754137 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x"] Oct 30 13:44:57.754606 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:57.754325 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" podUID="78b88187-8364-42ca-84dd-9e3d06c1ee3b" containerName="istio-proxy" containerID="cri-o://d6451422ee5b90a5a8b801460ce95e3d8f45936d165e010a7bb3ae06ada2a724" gracePeriod=30 Oct 30 13:44:58.133293 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:44:58.133258 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" podUID="246d0d04-97d3-4762-9075-777053be1de9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:44:59.402001 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:44:59.401961 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:45:03.084618 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.084578 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:45:03.106859 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.106821 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-data\") pod \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " Oct 30 13:45:03.107124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.106875 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b88187-8364-42ca-84dd-9e3d06c1ee3b-kube-api-access-z9g65\" (UniqueName: \"kubernetes.io/projected/78b88187-8364-42ca-84dd-9e3d06c1ee3b-kube-api-access-z9g65\") pod \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " Oct 30 13:45:03.107124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.106915 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-envoy\") pod \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " Oct 30 13:45:03.107124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.106983 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-workload-certs\") pod \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " Oct 30 13:45:03.107124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.107041 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-workload-socket\") pod \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " Oct 30 13:45:03.107124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.107087 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-podinfo\") pod \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " Oct 30 13:45:03.107124 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.107114 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istiod-ca-cert\") pod \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " Oct 30 13:45:03.107530 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.107144 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-data" pod "78b88187-8364-42ca-84dd-9e3d06c1ee3b" (UID: "78b88187-8364-42ca-84dd-9e3d06c1ee3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:03.107530 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.107232 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-credential-socket\") pod \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " Oct 30 13:45:03.107530 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.107311 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-token\" (UniqueName: \"kubernetes.io/projected/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-token\") pod \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\" (UID: \"78b88187-8364-42ca-84dd-9e3d06c1ee3b\") " Oct 30 13:45:03.107530 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.107511 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istiod-ca-cert" pod "78b88187-8364-42ca-84dd-9e3d06c1ee3b" (UID: "78b88187-8364-42ca-84dd-9e3d06c1ee3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:03.107765 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.107480 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-credential-socket" pod "78b88187-8364-42ca-84dd-9e3d06c1ee3b" (UID: "78b88187-8364-42ca-84dd-9e3d06c1ee3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:03.107765 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.107567 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-workload-certs" pod "78b88187-8364-42ca-84dd-9e3d06c1ee3b" (UID: "78b88187-8364-42ca-84dd-9e3d06c1ee3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:03.107765 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.107703 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-workload-socket" pod "78b88187-8364-42ca-84dd-9e3d06c1ee3b" (UID: "78b88187-8364-42ca-84dd-9e3d06c1ee3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:03.107931 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.107901 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.107931 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.107921 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.108090 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.107934 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.108090 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.107947 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.108090 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.107965 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.110217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.110144 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-envoy" pod "78b88187-8364-42ca-84dd-9e3d06c1ee3b" (UID: "78b88187-8364-42ca-84dd-9e3d06c1ee3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:03.110217 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.110200 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-podinfo" pod "78b88187-8364-42ca-84dd-9e3d06c1ee3b" (UID: "78b88187-8364-42ca-84dd-9e3d06c1ee3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:45:03.110414 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.110293 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-token" pod "78b88187-8364-42ca-84dd-9e3d06c1ee3b" (UID: "78b88187-8364-42ca-84dd-9e3d06c1ee3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:03.110414 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.110319 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b88187-8364-42ca-84dd-9e3d06c1ee3b-kube-api-access-z9g65" pod "78b88187-8364-42ca-84dd-9e3d06c1ee3b" (UID: "78b88187-8364-42ca-84dd-9e3d06c1ee3b"). InnerVolumeSpecName "kube-api-access-z9g65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:03.210498 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.209324 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9g65\" (UniqueName: \"kubernetes.io/projected/78b88187-8364-42ca-84dd-9e3d06c1ee3b-kube-api-access-z9g65\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.210498 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.209365 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.210498 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.209380 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.210498 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.209391 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b88187-8364-42ca-84dd-9e3d06c1ee3b-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:03.312825 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.312719 7010 generic.go:356] "Generic (PLEG): container finished" podID="78b88187-8364-42ca-84dd-9e3d06c1ee3b" containerID="d6451422ee5b90a5a8b801460ce95e3d8f45936d165e010a7bb3ae06ada2a724" exitCode=0 Oct 30 13:45:03.312825 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.312773 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" Oct 30 13:45:03.312825 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.312804 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" event={"ID":"78b88187-8364-42ca-84dd-9e3d06c1ee3b","Type":"ContainerDied","Data":"d6451422ee5b90a5a8b801460ce95e3d8f45936d165e010a7bb3ae06ada2a724"} Oct 30 13:45:03.313122 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.312894 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x" event={"ID":"78b88187-8364-42ca-84dd-9e3d06c1ee3b","Type":"ContainerDied","Data":"93ba4decb137977f943588f4595d32a04ab5ce233ec6643b7b33e62b94d78766"} Oct 30 13:45:03.313122 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.312972 7010 scope.go:122] "RemoveContainer" containerID="d6451422ee5b90a5a8b801460ce95e3d8f45936d165e010a7bb3ae06ada2a724" Oct 30 13:45:03.323876 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.323856 7010 scope.go:122] "RemoveContainer" containerID="d6451422ee5b90a5a8b801460ce95e3d8f45936d165e010a7bb3ae06ada2a724" Oct 30 13:45:03.324181 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:03.324159 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6451422ee5b90a5a8b801460ce95e3d8f45936d165e010a7bb3ae06ada2a724\": container with ID starting with d6451422ee5b90a5a8b801460ce95e3d8f45936d165e010a7bb3ae06ada2a724 not found: ID does not exist" containerID="d6451422ee5b90a5a8b801460ce95e3d8f45936d165e010a7bb3ae06ada2a724" Oct 30 13:45:03.324237 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.324190 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6451422ee5b90a5a8b801460ce95e3d8f45936d165e010a7bb3ae06ada2a724"} err="failed to get container status \"d6451422ee5b90a5a8b801460ce95e3d8f45936d165e010a7bb3ae06ada2a724\": rpc error: code = NotFound desc = could not find container \"d6451422ee5b90a5a8b801460ce95e3d8f45936d165e010a7bb3ae06ada2a724\": container with ID starting with d6451422ee5b90a5a8b801460ce95e3d8f45936d165e010a7bb3ae06ada2a724 not found: ID does not exist" Oct 30 13:45:03.369806 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.369765 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x"] Oct 30 13:45:03.374434 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.374406 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d833bd-openshift-default-bffbd4f5f-sjh9x"] Oct 30 13:45:03.488808 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.488778 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78b88187-8364-42ca-84dd-9e3d06c1ee3b" path="/var/lib/kubelet/pods/78b88187-8364-42ca-84dd-9e3d06c1ee3b/volumes" Oct 30 13:45:03.949950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.949904 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" containerID="cri-o://723b5951e3d67519bbe9b7a3111ac0a12aa965282675e1d04a9e85dc7f00890f" gracePeriod=45 Oct 30 13:45:03.959298 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:03.958988 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" containerID="cri-o://9c3a7e26342b79b16ef4a54c56e15e5681d27a802447cffcaaa9867bc46408e7" gracePeriod=45 Oct 30 13:45:04.001424 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.001386 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": dial tcp 10.134.0.39:8000: connect: connection refused" Oct 30 13:45:04.105844 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.105810 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" Oct 30 13:45:04.120118 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.120072 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/246d0d04-97d3-4762-9075-777053be1de9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246d0d04-97d3-4762-9075-777053be1de9-plugins-config-volume\") pod \"246d0d04-97d3-4762-9075-777053be1de9\" (UID: \"246d0d04-97d3-4762-9075-777053be1de9\") " Oct 30 13:45:04.120288 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.120206 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246d0d04-97d3-4762-9075-777053be1de9-kube-api-access-kfw9q\" (UniqueName: \"kubernetes.io/projected/246d0d04-97d3-4762-9075-777053be1de9-kube-api-access-kfw9q\") pod \"246d0d04-97d3-4762-9075-777053be1de9\" (UID: \"246d0d04-97d3-4762-9075-777053be1de9\") " Oct 30 13:45:04.121406 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.121374 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246d0d04-97d3-4762-9075-777053be1de9-plugins-config-volume" pod "246d0d04-97d3-4762-9075-777053be1de9" (UID: "246d0d04-97d3-4762-9075-777053be1de9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:04.122827 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.122793 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246d0d04-97d3-4762-9075-777053be1de9-kube-api-access-kfw9q" pod "246d0d04-97d3-4762-9075-777053be1de9" (UID: "246d0d04-97d3-4762-9075-777053be1de9"). InnerVolumeSpecName "kube-api-access-kfw9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:04.221626 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.221586 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfw9q\" (UniqueName: \"kubernetes.io/projected/246d0d04-97d3-4762-9075-777053be1de9-kube-api-access-kfw9q\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.221626 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.221625 7010 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246d0d04-97d3-4762-9075-777053be1de9-plugins-config-volume\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.224140 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.224112 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:45:04.234859 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.234834 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:45:04.322129 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.322021 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-model-cache\") pod \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " Oct 30 13:45:04.322129 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.322063 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-dshm\") pod \"7393071a-3009-43b4-afad-0ba041babef4\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " Oct 30 13:45:04.322129 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.322094 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-tls-certs\") pod \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " Oct 30 13:45:04.322129 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.322129 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-dshm\") pod \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " Oct 30 13:45:04.322646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.322172 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-home\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-home\") pod \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " Oct 30 13:45:04.322646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.322209 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7393071a-3009-43b4-afad-0ba041babef4-kube-api-access-jb9ww\" (UniqueName: \"kubernetes.io/projected/7393071a-3009-43b4-afad-0ba041babef4-kube-api-access-jb9ww\") pod \"7393071a-3009-43b4-afad-0ba041babef4\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " Oct 30 13:45:04.322646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.322270 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-kube-api-access-pdlhs\" (UniqueName: \"kubernetes.io/projected/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-kube-api-access-pdlhs\") pod \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " Oct 30 13:45:04.322646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.322324 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7393071a-3009-43b4-afad-0ba041babef4-tls-certs\" (UniqueName: \"kubernetes.io/secret/7393071a-3009-43b4-afad-0ba041babef4-tls-certs\") pod \"7393071a-3009-43b4-afad-0ba041babef4\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " Oct 30 13:45:04.322646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.322349 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-model-cache" pod "b8ceac9f-ae34-46f7-b04d-7d1484f880d7" (UID: "b8ceac9f-ae34-46f7-b04d-7d1484f880d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:04.322646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.322359 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-tmp-dir\") pod \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\" (UID: \"b8ceac9f-ae34-46f7-b04d-7d1484f880d7\") " Oct 30 13:45:04.322646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.322440 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-model-cache\") pod \"7393071a-3009-43b4-afad-0ba041babef4\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " Oct 30 13:45:04.322646 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.322615 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-tmp-dir" pod "b8ceac9f-ae34-46f7-b04d-7d1484f880d7" (UID: "b8ceac9f-ae34-46f7-b04d-7d1484f880d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:04.323402 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.322654 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-home\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-home\") pod \"7393071a-3009-43b4-afad-0ba041babef4\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " Oct 30 13:45:04.323402 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.322714 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-tmp-dir\") pod \"7393071a-3009-43b4-afad-0ba041babef4\" (UID: \"7393071a-3009-43b4-afad-0ba041babef4\") " Oct 30 13:45:04.323402 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.323130 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-home" pod "b8ceac9f-ae34-46f7-b04d-7d1484f880d7" (UID: "b8ceac9f-ae34-46f7-b04d-7d1484f880d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:04.323402 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.323237 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.323402 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.323257 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.323402 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.323271 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.324381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.323512 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-home" pod "7393071a-3009-43b4-afad-0ba041babef4" (UID: "7393071a-3009-43b4-afad-0ba041babef4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:04.324381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.323743 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-model-cache" pod "7393071a-3009-43b4-afad-0ba041babef4" (UID: "7393071a-3009-43b4-afad-0ba041babef4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:04.324381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.323879 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-tmp-dir" pod "7393071a-3009-43b4-afad-0ba041babef4" (UID: "7393071a-3009-43b4-afad-0ba041babef4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:04.324381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.324133 7010 generic.go:356] "Generic (PLEG): container finished" podID="246d0d04-97d3-4762-9075-777053be1de9" containerID="0a6b54b305cb6edd985e73c919029d7d549cec5825798bf02070a3330b0b864d" exitCode=0 Oct 30 13:45:04.324381 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.324285 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" Oct 30 13:45:04.325189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.324882 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" event={"ID":"246d0d04-97d3-4762-9075-777053be1de9","Type":"ContainerDied","Data":"0a6b54b305cb6edd985e73c919029d7d549cec5825798bf02070a3330b0b864d"} Oct 30 13:45:04.325189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.324947 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6" event={"ID":"246d0d04-97d3-4762-9075-777053be1de9","Type":"ContainerDied","Data":"7e2d6a49479cb25e039f207c50e4136d93f674e7478c92454fcc69171c35d2e6"} Oct 30 13:45:04.325189 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.324978 7010 scope.go:122] "RemoveContainer" containerID="0a6b54b305cb6edd985e73c919029d7d549cec5825798bf02070a3330b0b864d" Oct 30 13:45:04.327100 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.327072 7010 generic.go:356] "Generic (PLEG): container finished" podID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerID="9c3a7e26342b79b16ef4a54c56e15e5681d27a802447cffcaaa9867bc46408e7" exitCode=0 Oct 30 13:45:04.327249 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.327229 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" event={"ID":"b8ceac9f-ae34-46f7-b04d-7d1484f880d7","Type":"ContainerDied","Data":"9c3a7e26342b79b16ef4a54c56e15e5681d27a802447cffcaaa9867bc46408e7"} Oct 30 13:45:04.327327 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.327311 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" event={"ID":"b8ceac9f-ae34-46f7-b04d-7d1484f880d7","Type":"ContainerDied","Data":"ee816364acf60d9a31c52f51c5dc7f7a678acf50771f62089a30b0470dbefa84"} Oct 30 13:45:04.327400 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.327385 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4" Oct 30 13:45:04.331333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.330896 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-dshm" pod "7393071a-3009-43b4-afad-0ba041babef4" (UID: "7393071a-3009-43b4-afad-0ba041babef4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:04.331333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.330976 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-tls-certs" pod "b8ceac9f-ae34-46f7-b04d-7d1484f880d7" (UID: "b8ceac9f-ae34-46f7-b04d-7d1484f880d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:04.331333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.330974 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7393071a-3009-43b4-afad-0ba041babef4-kube-api-access-jb9ww" pod "7393071a-3009-43b4-afad-0ba041babef4" (UID: "7393071a-3009-43b4-afad-0ba041babef4"). InnerVolumeSpecName "kube-api-access-jb9ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:04.331333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.331036 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-dshm" pod "b8ceac9f-ae34-46f7-b04d-7d1484f880d7" (UID: "b8ceac9f-ae34-46f7-b04d-7d1484f880d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:04.331333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.331165 7010 generic.go:356] "Generic (PLEG): container finished" podID="7393071a-3009-43b4-afad-0ba041babef4" containerID="723b5951e3d67519bbe9b7a3111ac0a12aa965282675e1d04a9e85dc7f00890f" exitCode=0 Oct 30 13:45:04.331333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.331197 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" event={"ID":"7393071a-3009-43b4-afad-0ba041babef4","Type":"ContainerDied","Data":"723b5951e3d67519bbe9b7a3111ac0a12aa965282675e1d04a9e85dc7f00890f"} Oct 30 13:45:04.331333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.331222 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" Oct 30 13:45:04.331333 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.331223 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk" event={"ID":"7393071a-3009-43b4-afad-0ba041babef4","Type":"ContainerDied","Data":"2c21bb674bbf417906446e9c658c11d62ca67f304546d15e346131299aca1090"} Oct 30 13:45:04.332716 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.332687 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-kube-api-access-pdlhs" pod "b8ceac9f-ae34-46f7-b04d-7d1484f880d7" (UID: "b8ceac9f-ae34-46f7-b04d-7d1484f880d7"). InnerVolumeSpecName "kube-api-access-pdlhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:04.332946 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.332916 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7393071a-3009-43b4-afad-0ba041babef4-tls-certs" pod "7393071a-3009-43b4-afad-0ba041babef4" (UID: "7393071a-3009-43b4-afad-0ba041babef4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:04.343207 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.343183 7010 scope.go:122] "RemoveContainer" containerID="0a6b54b305cb6edd985e73c919029d7d549cec5825798bf02070a3330b0b864d" Oct 30 13:45:04.343811 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:04.343774 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6b54b305cb6edd985e73c919029d7d549cec5825798bf02070a3330b0b864d\": container with ID starting with 0a6b54b305cb6edd985e73c919029d7d549cec5825798bf02070a3330b0b864d not found: ID does not exist" containerID="0a6b54b305cb6edd985e73c919029d7d549cec5825798bf02070a3330b0b864d" Oct 30 13:45:04.343976 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.343845 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6b54b305cb6edd985e73c919029d7d549cec5825798bf02070a3330b0b864d"} err="failed to get container status \"0a6b54b305cb6edd985e73c919029d7d549cec5825798bf02070a3330b0b864d\": rpc error: code = NotFound desc = could not find container \"0a6b54b305cb6edd985e73c919029d7d549cec5825798bf02070a3330b0b864d\": container with ID starting with 0a6b54b305cb6edd985e73c919029d7d549cec5825798bf02070a3330b0b864d not found: ID does not exist" Oct 30 13:45:04.343976 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.343872 7010 scope.go:122] "RemoveContainer" containerID="9c3a7e26342b79b16ef4a54c56e15e5681d27a802447cffcaaa9867bc46408e7" Oct 30 13:45:04.355706 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.355632 7010 scope.go:122] "RemoveContainer" containerID="9c3a7e26342b79b16ef4a54c56e15e5681d27a802447cffcaaa9867bc46408e7" Oct 30 13:45:04.356076 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:04.356049 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c3a7e26342b79b16ef4a54c56e15e5681d27a802447cffcaaa9867bc46408e7\": container with ID starting with 9c3a7e26342b79b16ef4a54c56e15e5681d27a802447cffcaaa9867bc46408e7 not found: ID does not exist" containerID="9c3a7e26342b79b16ef4a54c56e15e5681d27a802447cffcaaa9867bc46408e7" Oct 30 13:45:04.356175 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.356087 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3a7e26342b79b16ef4a54c56e15e5681d27a802447cffcaaa9867bc46408e7"} err="failed to get container status \"9c3a7e26342b79b16ef4a54c56e15e5681d27a802447cffcaaa9867bc46408e7\": rpc error: code = NotFound desc = could not find container \"9c3a7e26342b79b16ef4a54c56e15e5681d27a802447cffcaaa9867bc46408e7\": container with ID starting with 9c3a7e26342b79b16ef4a54c56e15e5681d27a802447cffcaaa9867bc46408e7 not found: ID does not exist" Oct 30 13:45:04.356175 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.356110 7010 scope.go:122] "RemoveContainer" containerID="723b5951e3d67519bbe9b7a3111ac0a12aa965282675e1d04a9e85dc7f00890f" Oct 30 13:45:04.366786 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.366758 7010 scope.go:122] "RemoveContainer" containerID="723b5951e3d67519bbe9b7a3111ac0a12aa965282675e1d04a9e85dc7f00890f" Oct 30 13:45:04.367175 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:04.367151 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"723b5951e3d67519bbe9b7a3111ac0a12aa965282675e1d04a9e85dc7f00890f\": container with ID starting with 723b5951e3d67519bbe9b7a3111ac0a12aa965282675e1d04a9e85dc7f00890f not found: ID does not exist" containerID="723b5951e3d67519bbe9b7a3111ac0a12aa965282675e1d04a9e85dc7f00890f" Oct 30 13:45:04.367263 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.367188 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723b5951e3d67519bbe9b7a3111ac0a12aa965282675e1d04a9e85dc7f00890f"} err="failed to get container status \"723b5951e3d67519bbe9b7a3111ac0a12aa965282675e1d04a9e85dc7f00890f\": rpc error: code = NotFound desc = could not find container \"723b5951e3d67519bbe9b7a3111ac0a12aa965282675e1d04a9e85dc7f00890f\": container with ID starting with 723b5951e3d67519bbe9b7a3111ac0a12aa965282675e1d04a9e85dc7f00890f not found: ID does not exist" Oct 30 13:45:04.424373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.424338 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.424373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.424369 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.424373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.424379 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.424627 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.424388 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.424627 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.424397 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.424627 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.424407 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb9ww\" (UniqueName: \"kubernetes.io/projected/7393071a-3009-43b4-afad-0ba041babef4-kube-api-access-jb9ww\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.424627 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.424415 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdlhs\" (UniqueName: \"kubernetes.io/projected/b8ceac9f-ae34-46f7-b04d-7d1484f880d7-kube-api-access-pdlhs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.424627 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.424424 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7393071a-3009-43b4-afad-0ba041babef4-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.424627 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.424432 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7393071a-3009-43b4-afad-0ba041babef4-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:04.433647 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.433612 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6"] Oct 30 13:45:04.438889 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.438859 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-8d82d5-7978d67f9f-2pxd6"] Oct 30 13:45:04.674910 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.674873 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4"] Oct 30 13:45:04.681436 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.681402 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/simulator-w5-e34fd5-kserve-7f97bd9c4c-wgrt4"] Oct 30 13:45:04.716690 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.716649 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk"] Oct 30 13:45:04.720529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:04.720470 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e34fd5/premium-w5-e34fd5-kserve-586b6b5d7d-s7mwk"] Oct 30 13:45:05.488588 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:05.488550 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="246d0d04-97d3-4762-9075-777053be1de9" path="/var/lib/kubelet/pods/246d0d04-97d3-4762-9075-777053be1de9/volumes" Oct 30 13:45:05.489103 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:05.489047 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7393071a-3009-43b4-afad-0ba041babef4" path="/var/lib/kubelet/pods/7393071a-3009-43b4-afad-0ba041babef4/volumes" Oct 30 13:45:05.489606 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:05.489585 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" path="/var/lib/kubelet/pods/b8ceac9f-ae34-46f7-b04d-7d1484f880d7/volumes" Oct 30 13:45:09.991288 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:09.991242 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" podUID="4e411b13-5f67-4954-bdcd-582026eb0ac4" containerName="main" containerID="cri-o://0dc982ca5f8dd20192867ca34a120912fb46dd809dcc43fb9bae1177baf6f1e4" gracePeriod=45 Oct 30 13:45:10.038991 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.038940 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" containerID="cri-o://6d8ebf7def231cd8c2263915af76780699089fb6ea2c591841c49117fe7afae3" gracePeriod=45 Oct 30 13:45:10.306242 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.306207 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:45:10.332323 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.332287 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:45:10.376706 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.376665 7010 generic.go:356] "Generic (PLEG): container finished" podID="4e411b13-5f67-4954-bdcd-582026eb0ac4" containerID="0dc982ca5f8dd20192867ca34a120912fb46dd809dcc43fb9bae1177baf6f1e4" exitCode=0 Oct 30 13:45:10.376890 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.376707 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" event={"ID":"4e411b13-5f67-4954-bdcd-582026eb0ac4","Type":"ContainerDied","Data":"0dc982ca5f8dd20192867ca34a120912fb46dd809dcc43fb9bae1177baf6f1e4"} Oct 30 13:45:10.376890 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.376728 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" Oct 30 13:45:10.376890 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.376764 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn" event={"ID":"4e411b13-5f67-4954-bdcd-582026eb0ac4","Type":"ContainerDied","Data":"6c9e2a49636a73f670536bfc6a2b75996d8b65d62c40e869003942e605ea37e0"} Oct 30 13:45:10.376890 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.376786 7010 scope.go:122] "RemoveContainer" containerID="0dc982ca5f8dd20192867ca34a120912fb46dd809dcc43fb9bae1177baf6f1e4" Oct 30 13:45:10.378530 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.378470 7010 generic.go:356] "Generic (PLEG): container finished" podID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerID="6d8ebf7def231cd8c2263915af76780699089fb6ea2c591841c49117fe7afae3" exitCode=0 Oct 30 13:45:10.378663 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.378538 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" Oct 30 13:45:10.378663 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.378539 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" event={"ID":"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9","Type":"ContainerDied","Data":"6d8ebf7def231cd8c2263915af76780699089fb6ea2c591841c49117fe7afae3"} Oct 30 13:45:10.378663 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.378582 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq" event={"ID":"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9","Type":"ContainerDied","Data":"44b18e5c4a0b39b917769c2bb1fc35f095ce963911a898802a795f0cea3f596a"} Oct 30 13:45:10.386571 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.386535 7010 scope.go:122] "RemoveContainer" containerID="0dc982ca5f8dd20192867ca34a120912fb46dd809dcc43fb9bae1177baf6f1e4" Oct 30 13:45:10.386895 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:10.386873 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc982ca5f8dd20192867ca34a120912fb46dd809dcc43fb9bae1177baf6f1e4\": container with ID starting with 0dc982ca5f8dd20192867ca34a120912fb46dd809dcc43fb9bae1177baf6f1e4 not found: ID does not exist" containerID="0dc982ca5f8dd20192867ca34a120912fb46dd809dcc43fb9bae1177baf6f1e4" Oct 30 13:45:10.386986 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.386903 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc982ca5f8dd20192867ca34a120912fb46dd809dcc43fb9bae1177baf6f1e4"} err="failed to get container status \"0dc982ca5f8dd20192867ca34a120912fb46dd809dcc43fb9bae1177baf6f1e4\": rpc error: code = NotFound desc = could not find container \"0dc982ca5f8dd20192867ca34a120912fb46dd809dcc43fb9bae1177baf6f1e4\": container with ID starting with 0dc982ca5f8dd20192867ca34a120912fb46dd809dcc43fb9bae1177baf6f1e4 not found: ID does not exist" Oct 30 13:45:10.386986 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.386922 7010 scope.go:122] "RemoveContainer" containerID="6d8ebf7def231cd8c2263915af76780699089fb6ea2c591841c49117fe7afae3" Oct 30 13:45:10.396529 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.396463 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-dshm\") pod \"4e411b13-5f67-4954-bdcd-582026eb0ac4\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " Oct 30 13:45:10.396762 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.396738 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-tls-certs\") pod \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " Oct 30 13:45:10.396841 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.396802 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-model-cache\") pod \"4e411b13-5f67-4954-bdcd-582026eb0ac4\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " Oct 30 13:45:10.396841 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.396830 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-tmp-dir\") pod \"4e411b13-5f67-4954-bdcd-582026eb0ac4\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " Oct 30 13:45:10.397075 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.396889 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-home\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-home\") pod \"4e411b13-5f67-4954-bdcd-582026eb0ac4\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " Oct 30 13:45:10.397075 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.396927 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-tmp-dir\") pod \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " Oct 30 13:45:10.397075 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.396971 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e411b13-5f67-4954-bdcd-582026eb0ac4-kube-api-access-2dbxm\" (UniqueName: \"kubernetes.io/projected/4e411b13-5f67-4954-bdcd-582026eb0ac4-kube-api-access-2dbxm\") pod \"4e411b13-5f67-4954-bdcd-582026eb0ac4\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " Oct 30 13:45:10.397075 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.397009 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-model-cache\") pod \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " Oct 30 13:45:10.397075 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.397047 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e411b13-5f67-4954-bdcd-582026eb0ac4-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e411b13-5f67-4954-bdcd-582026eb0ac4-tls-certs\") pod \"4e411b13-5f67-4954-bdcd-582026eb0ac4\" (UID: \"4e411b13-5f67-4954-bdcd-582026eb0ac4\") " Oct 30 13:45:10.397351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.397077 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-dshm\") pod \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " Oct 30 13:45:10.397351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.397093 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-model-cache" pod "4e411b13-5f67-4954-bdcd-582026eb0ac4" (UID: "4e411b13-5f67-4954-bdcd-582026eb0ac4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:10.397351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.397106 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-kube-api-access-pd75f\" (UniqueName: \"kubernetes.io/projected/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-kube-api-access-pd75f\") pod \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " Oct 30 13:45:10.397351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.397135 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-home\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-home\") pod \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\" (UID: \"e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9\") " Oct 30 13:45:10.397351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.397232 7010 scope.go:122] "RemoveContainer" containerID="6d8ebf7def231cd8c2263915af76780699089fb6ea2c591841c49117fe7afae3" Oct 30 13:45:10.397692 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.397658 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.397957 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.397841 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-home" pod "4e411b13-5f67-4954-bdcd-582026eb0ac4" (UID: "4e411b13-5f67-4954-bdcd-582026eb0ac4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:10.397957 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.397867 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-tmp-dir" pod "4e411b13-5f67-4954-bdcd-582026eb0ac4" (UID: "4e411b13-5f67-4954-bdcd-582026eb0ac4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:10.397957 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.397885 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-home" pod "e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" (UID: "e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:10.398149 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.398066 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-model-cache" pod "e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" (UID: "e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:10.398149 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:10.398104 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d8ebf7def231cd8c2263915af76780699089fb6ea2c591841c49117fe7afae3\": container with ID starting with 6d8ebf7def231cd8c2263915af76780699089fb6ea2c591841c49117fe7afae3 not found: ID does not exist" containerID="6d8ebf7def231cd8c2263915af76780699089fb6ea2c591841c49117fe7afae3" Oct 30 13:45:10.398303 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.398147 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d8ebf7def231cd8c2263915af76780699089fb6ea2c591841c49117fe7afae3"} err="failed to get container status \"6d8ebf7def231cd8c2263915af76780699089fb6ea2c591841c49117fe7afae3\": rpc error: code = NotFound desc = could not find container \"6d8ebf7def231cd8c2263915af76780699089fb6ea2c591841c49117fe7afae3\": container with ID starting with 6d8ebf7def231cd8c2263915af76780699089fb6ea2c591841c49117fe7afae3 not found: ID does not exist" Oct 30 13:45:10.398373 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.398355 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-tmp-dir" pod "e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" (UID: "e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:10.399922 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.399887 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-tls-certs" pod "e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" (UID: "e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:10.403395 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.400730 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e411b13-5f67-4954-bdcd-582026eb0ac4-kube-api-access-2dbxm" pod "4e411b13-5f67-4954-bdcd-582026eb0ac4" (UID: "4e411b13-5f67-4954-bdcd-582026eb0ac4"). InnerVolumeSpecName "kube-api-access-2dbxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:10.405892 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.405860 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-kube-api-access-pd75f" pod "e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" (UID: "e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9"). InnerVolumeSpecName "kube-api-access-pd75f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:10.407382 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.407344 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-dshm" pod "e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" (UID: "e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:10.408696 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.408657 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e411b13-5f67-4954-bdcd-582026eb0ac4-tls-certs" pod "4e411b13-5f67-4954-bdcd-582026eb0ac4" (UID: "4e411b13-5f67-4954-bdcd-582026eb0ac4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:10.409029 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.409003 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-dshm" pod "4e411b13-5f67-4954-bdcd-582026eb0ac4" (UID: "4e411b13-5f67-4954-bdcd-582026eb0ac4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:10.499145 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.499066 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.499145 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.499095 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.499145 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.499106 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.499145 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.499113 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e411b13-5f67-4954-bdcd-582026eb0ac4-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.499145 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.499121 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.499145 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.499129 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dbxm\" (UniqueName: \"kubernetes.io/projected/4e411b13-5f67-4954-bdcd-582026eb0ac4-kube-api-access-2dbxm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.499145 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.499139 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.499145 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.499149 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e411b13-5f67-4954-bdcd-582026eb0ac4-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.499539 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.499158 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.499539 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.499166 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd75f\" (UniqueName: \"kubernetes.io/projected/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-kube-api-access-pd75f\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.499539 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.499174 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:10.723933 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.723895 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn"] Oct 30 13:45:10.729303 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.729273 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/distinct-2-w1-52a649-kserve-6df466ccfb-8hdvn"] Oct 30 13:45:10.766106 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.766008 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq"] Oct 30 13:45:10.771356 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:10.771323 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-52a649/simulator-w1-52a649-kserve-5b9b5777-drfjq"] Oct 30 13:45:11.487882 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:11.487842 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e411b13-5f67-4954-bdcd-582026eb0ac4" path="/var/lib/kubelet/pods/4e411b13-5f67-4954-bdcd-582026eb0ac4/volumes" Oct 30 13:45:11.488290 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:11.488268 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" path="/var/lib/kubelet/pods/e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9/volumes" Oct 30 13:45:11.605845 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:11.605802 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49"] Oct 30 13:45:11.606046 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:11.606028 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" podUID="4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" containerName="istio-proxy" containerID="cri-o://8522aa41778bc42d1c0dcab71bd829765582c89ac4da7d3a836ba8ae71ba4408" gracePeriod=30 Oct 30 13:45:13.326165 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:13.326125 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg"] Oct 30 13:45:13.326650 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:13.326372 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" podUID="37fbf9d2-f376-4f7d-a514-54e51342f2f6" containerName="maas-api" containerID="cri-o://211ef12859be642ed0c067b0ff4e1df7759429b9ac9c65b0f23a3190186fe54d" gracePeriod=30 Oct 30 13:45:13.589289 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:13.589216 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:45:13.735135 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:13.735103 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fbf9d2-f376-4f7d-a514-54e51342f2f6-kube-api-access-8vt9s\" (UniqueName: \"kubernetes.io/projected/37fbf9d2-f376-4f7d-a514-54e51342f2f6-kube-api-access-8vt9s\") pod \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\" (UID: \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\") " Oct 30 13:45:13.735324 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:13.735153 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37fbf9d2-f376-4f7d-a514-54e51342f2f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37fbf9d2-f376-4f7d-a514-54e51342f2f6-maas-api-tls\") pod \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\" (UID: \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\") " Oct 30 13:45:13.735324 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:13.735187 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37fbf9d2-f376-4f7d-a514-54e51342f2f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/37fbf9d2-f376-4f7d-a514-54e51342f2f6-metrics-certs\") pod \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\" (UID: \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\") " Oct 30 13:45:13.735324 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:13.735248 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37fbf9d2-f376-4f7d-a514-54e51342f2f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37fbf9d2-f376-4f7d-a514-54e51342f2f6-openshift-service-ca\") pod \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\" (UID: \"37fbf9d2-f376-4f7d-a514-54e51342f2f6\") " Oct 30 13:45:13.735739 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:13.735707 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37fbf9d2-f376-4f7d-a514-54e51342f2f6-openshift-service-ca" pod "37fbf9d2-f376-4f7d-a514-54e51342f2f6" (UID: "37fbf9d2-f376-4f7d-a514-54e51342f2f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:13.737426 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:13.737392 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37fbf9d2-f376-4f7d-a514-54e51342f2f6-metrics-certs" pod "37fbf9d2-f376-4f7d-a514-54e51342f2f6" (UID: "37fbf9d2-f376-4f7d-a514-54e51342f2f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:13.737595 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:13.737480 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fbf9d2-f376-4f7d-a514-54e51342f2f6-kube-api-access-8vt9s" pod "37fbf9d2-f376-4f7d-a514-54e51342f2f6" (UID: "37fbf9d2-f376-4f7d-a514-54e51342f2f6"). InnerVolumeSpecName "kube-api-access-8vt9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:13.737595 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:13.737584 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37fbf9d2-f376-4f7d-a514-54e51342f2f6-maas-api-tls" pod "37fbf9d2-f376-4f7d-a514-54e51342f2f6" (UID: "37fbf9d2-f376-4f7d-a514-54e51342f2f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:13.836582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:13.836542 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vt9s\" (UniqueName: \"kubernetes.io/projected/37fbf9d2-f376-4f7d-a514-54e51342f2f6-kube-api-access-8vt9s\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:13.836582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:13.836575 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37fbf9d2-f376-4f7d-a514-54e51342f2f6-maas-api-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:13.836582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:13.836584 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37fbf9d2-f376-4f7d-a514-54e51342f2f6-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:13.836820 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:13.836593 7010 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37fbf9d2-f376-4f7d-a514-54e51342f2f6-openshift-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:14.412126 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:14.412093 7010 generic.go:356] "Generic (PLEG): container finished" podID="37fbf9d2-f376-4f7d-a514-54e51342f2f6" containerID="211ef12859be642ed0c067b0ff4e1df7759429b9ac9c65b0f23a3190186fe54d" exitCode=0 Oct 30 13:45:14.412610 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:14.412140 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" Oct 30 13:45:14.412610 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:14.412170 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" event={"ID":"37fbf9d2-f376-4f7d-a514-54e51342f2f6","Type":"ContainerDied","Data":"211ef12859be642ed0c067b0ff4e1df7759429b9ac9c65b0f23a3190186fe54d"} Oct 30 13:45:14.412610 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:14.412222 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg" event={"ID":"37fbf9d2-f376-4f7d-a514-54e51342f2f6","Type":"ContainerDied","Data":"4020d227e099815e1908bef9efcd9efa9e18deb54ffda7923d6f2f700275f6ac"} Oct 30 13:45:14.412610 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:14.412245 7010 scope.go:122] "RemoveContainer" containerID="211ef12859be642ed0c067b0ff4e1df7759429b9ac9c65b0f23a3190186fe54d" Oct 30 13:45:14.421752 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:14.421730 7010 scope.go:122] "RemoveContainer" containerID="211ef12859be642ed0c067b0ff4e1df7759429b9ac9c65b0f23a3190186fe54d" Oct 30 13:45:14.422041 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:14.422022 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211ef12859be642ed0c067b0ff4e1df7759429b9ac9c65b0f23a3190186fe54d\": container with ID starting with 211ef12859be642ed0c067b0ff4e1df7759429b9ac9c65b0f23a3190186fe54d not found: ID does not exist" containerID="211ef12859be642ed0c067b0ff4e1df7759429b9ac9c65b0f23a3190186fe54d" Oct 30 13:45:14.422107 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:14.422054 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211ef12859be642ed0c067b0ff4e1df7759429b9ac9c65b0f23a3190186fe54d"} err="failed to get container status \"211ef12859be642ed0c067b0ff4e1df7759429b9ac9c65b0f23a3190186fe54d\": rpc error: code = NotFound desc = could not find container \"211ef12859be642ed0c067b0ff4e1df7759429b9ac9c65b0f23a3190186fe54d\": container with ID starting with 211ef12859be642ed0c067b0ff4e1df7759429b9ac9c65b0f23a3190186fe54d not found: ID does not exist" Oct 30 13:45:14.468560 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:14.468523 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg"] Oct 30 13:45:14.481134 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:14.481103 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f07dfe-7bdf6855d8-9zpmg"] Oct 30 13:45:15.488516 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:15.488461 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37fbf9d2-f376-4f7d-a514-54e51342f2f6" path="/var/lib/kubelet/pods/37fbf9d2-f376-4f7d-a514-54e51342f2f6/volumes" Oct 30 13:45:15.916020 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:15.915982 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn"] Oct 30 13:45:15.916284 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:15.916256 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" podUID="d1350a65-bf30-427d-be17-0ad9bcf8aa0a" containerName="maas-api" containerID="cri-o://79745f9109309b212457cdde1bd8b1755584e723aca85380993a494845c9e19e" gracePeriod=30 Oct 30 13:45:16.218604 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.218567 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:45:16.366072 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.366038 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-maas-api-tls\") pod \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\" (UID: \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\") " Oct 30 13:45:16.366072 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.366091 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-metrics-certs\") pod \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\" (UID: \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\") " Oct 30 13:45:16.366322 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.366213 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-kube-api-access-66rqp\" (UniqueName: \"kubernetes.io/projected/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-kube-api-access-66rqp\") pod \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\" (UID: \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\") " Oct 30 13:45:16.366366 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.366326 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-openshift-service-ca\") pod \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\" (UID: \"d1350a65-bf30-427d-be17-0ad9bcf8aa0a\") " Oct 30 13:45:16.366812 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.366774 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-openshift-service-ca" pod "d1350a65-bf30-427d-be17-0ad9bcf8aa0a" (UID: "d1350a65-bf30-427d-be17-0ad9bcf8aa0a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:16.368362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.368329 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-metrics-certs" pod "d1350a65-bf30-427d-be17-0ad9bcf8aa0a" (UID: "d1350a65-bf30-427d-be17-0ad9bcf8aa0a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:16.368362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.368354 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-maas-api-tls" pod "d1350a65-bf30-427d-be17-0ad9bcf8aa0a" (UID: "d1350a65-bf30-427d-be17-0ad9bcf8aa0a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:16.368514 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.368332 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-kube-api-access-66rqp" pod "d1350a65-bf30-427d-be17-0ad9bcf8aa0a" (UID: "d1350a65-bf30-427d-be17-0ad9bcf8aa0a"). InnerVolumeSpecName "kube-api-access-66rqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:16.426321 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.426226 7010 generic.go:356] "Generic (PLEG): container finished" podID="d1350a65-bf30-427d-be17-0ad9bcf8aa0a" containerID="79745f9109309b212457cdde1bd8b1755584e723aca85380993a494845c9e19e" exitCode=0 Oct 30 13:45:16.426321 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.426274 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" Oct 30 13:45:16.426321 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.426290 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" event={"ID":"d1350a65-bf30-427d-be17-0ad9bcf8aa0a","Type":"ContainerDied","Data":"79745f9109309b212457cdde1bd8b1755584e723aca85380993a494845c9e19e"} Oct 30 13:45:16.426583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.426367 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn" event={"ID":"d1350a65-bf30-427d-be17-0ad9bcf8aa0a","Type":"ContainerDied","Data":"0c0fcea8bf29a7614d0b5788468c3054ba4f0aa7cdd3725dca605365dd4c3558"} Oct 30 13:45:16.426583 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.426391 7010 scope.go:122] "RemoveContainer" containerID="79745f9109309b212457cdde1bd8b1755584e723aca85380993a494845c9e19e" Oct 30 13:45:16.437338 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.437319 7010 scope.go:122] "RemoveContainer" containerID="79745f9109309b212457cdde1bd8b1755584e723aca85380993a494845c9e19e" Oct 30 13:45:16.437738 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:16.437715 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79745f9109309b212457cdde1bd8b1755584e723aca85380993a494845c9e19e\": container with ID starting with 79745f9109309b212457cdde1bd8b1755584e723aca85380993a494845c9e19e not found: ID does not exist" containerID="79745f9109309b212457cdde1bd8b1755584e723aca85380993a494845c9e19e" Oct 30 13:45:16.437809 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.437752 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79745f9109309b212457cdde1bd8b1755584e723aca85380993a494845c9e19e"} err="failed to get container status \"79745f9109309b212457cdde1bd8b1755584e723aca85380993a494845c9e19e\": rpc error: code = NotFound desc = could not find container \"79745f9109309b212457cdde1bd8b1755584e723aca85380993a494845c9e19e\": container with ID starting with 79745f9109309b212457cdde1bd8b1755584e723aca85380993a494845c9e19e not found: ID does not exist" Oct 30 13:45:16.467623 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.467580 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:16.467623 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.467613 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66rqp\" (UniqueName: \"kubernetes.io/projected/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-kube-api-access-66rqp\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:16.467623 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.467624 7010 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-openshift-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:16.467623 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.467633 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1350a65-bf30-427d-be17-0ad9bcf8aa0a-maas-api-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:16.479006 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.478973 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn"] Oct 30 13:45:16.484052 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.484013 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-52a649-f4946f6bb-9kftn"] Oct 30 13:45:16.862319 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.862285 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:45:16.972951 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.972910 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-data\") pod \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " Oct 30 13:45:16.973153 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.972985 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-workload-socket\") pod \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " Oct 30 13:45:16.973153 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.973038 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-kube-api-access-vj86n\" (UniqueName: \"kubernetes.io/projected/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-kube-api-access-vj86n\") pod \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " Oct 30 13:45:16.973153 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.973138 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-token\" (UniqueName: \"kubernetes.io/projected/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-token\") pod \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " Oct 30 13:45:16.973342 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.973191 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istiod-ca-cert\") pod \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " Oct 30 13:45:16.973342 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.973219 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-credential-socket\") pod \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " Oct 30 13:45:16.973342 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.973246 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-envoy\") pod \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " Oct 30 13:45:16.973342 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.973293 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-podinfo\") pod \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " Oct 30 13:45:16.973342 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.973241 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-data" pod "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" (UID: "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:16.973342 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.973255 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-workload-socket" pod "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" (UID: "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:16.973726 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.973352 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-workload-certs\") pod \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\" (UID: \"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2\") " Oct 30 13:45:16.973726 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.973532 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-credential-socket" pod "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" (UID: "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:16.973726 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.973614 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istiod-ca-cert" pod "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" (UID: "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:16.973879 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.973746 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-workload-certs" pod "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" (UID: "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:16.973879 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.973852 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:16.973879 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.973874 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:16.974041 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.973887 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:16.974041 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.973901 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:16.974041 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.973912 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:16.975569 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.975501 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-podinfo" pod "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" (UID: "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:45:16.975569 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.975558 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-envoy" pod "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" (UID: "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:16.975747 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.975573 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-kube-api-access-vj86n" pod "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" (UID: "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2"). InnerVolumeSpecName "kube-api-access-vj86n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:16.975747 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:16.975661 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-token" pod "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" (UID: "4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:17.042660 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.042603 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt"] Oct 30 13:45:17.055708 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.055666 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2"] Oct 30 13:45:17.064135 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.064096 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp"] Oct 30 13:45:17.075064 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.075033 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj86n\" (UniqueName: \"kubernetes.io/projected/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-kube-api-access-vj86n\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:17.075064 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.075062 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:17.075064 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.075071 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:17.075382 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.075079 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:17.086994 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.086960 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt"] Oct 30 13:45:17.435029 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.434992 7010 generic.go:356] "Generic (PLEG): container finished" podID="4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" containerID="8522aa41778bc42d1c0dcab71bd829765582c89ac4da7d3a836ba8ae71ba4408" exitCode=0 Oct 30 13:45:17.435029 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.435031 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" event={"ID":"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2","Type":"ContainerDied","Data":"8522aa41778bc42d1c0dcab71bd829765582c89ac4da7d3a836ba8ae71ba4408"} Oct 30 13:45:17.435306 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.435053 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" Oct 30 13:45:17.435306 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.435072 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49" event={"ID":"4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2","Type":"ContainerDied","Data":"9f2ac25441e81cecfcb7c148a0eadc29f0a1bd6e747908624cd275d249d52ec0"} Oct 30 13:45:17.435306 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.435095 7010 scope.go:122] "RemoveContainer" containerID="8522aa41778bc42d1c0dcab71bd829765582c89ac4da7d3a836ba8ae71ba4408" Oct 30 13:45:17.445709 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.445688 7010 scope.go:122] "RemoveContainer" containerID="d7ae7821917dc2438911ded9942372ca66f915b8ef34434bf5aafac3c9d0a874" Oct 30 13:45:17.456087 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.456064 7010 scope.go:122] "RemoveContainer" containerID="8522aa41778bc42d1c0dcab71bd829765582c89ac4da7d3a836ba8ae71ba4408" Oct 30 13:45:17.456460 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:17.456438 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8522aa41778bc42d1c0dcab71bd829765582c89ac4da7d3a836ba8ae71ba4408\": container with ID starting with 8522aa41778bc42d1c0dcab71bd829765582c89ac4da7d3a836ba8ae71ba4408 not found: ID does not exist" containerID="8522aa41778bc42d1c0dcab71bd829765582c89ac4da7d3a836ba8ae71ba4408" Oct 30 13:45:17.456523 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.456471 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8522aa41778bc42d1c0dcab71bd829765582c89ac4da7d3a836ba8ae71ba4408"} err="failed to get container status \"8522aa41778bc42d1c0dcab71bd829765582c89ac4da7d3a836ba8ae71ba4408\": rpc error: code = NotFound desc = could not find container \"8522aa41778bc42d1c0dcab71bd829765582c89ac4da7d3a836ba8ae71ba4408\": container with ID starting with 8522aa41778bc42d1c0dcab71bd829765582c89ac4da7d3a836ba8ae71ba4408 not found: ID does not exist" Oct 30 13:45:17.456523 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.456512 7010 scope.go:122] "RemoveContainer" containerID="d7ae7821917dc2438911ded9942372ca66f915b8ef34434bf5aafac3c9d0a874" Oct 30 13:45:17.456853 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:17.456828 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ae7821917dc2438911ded9942372ca66f915b8ef34434bf5aafac3c9d0a874\": container with ID starting with d7ae7821917dc2438911ded9942372ca66f915b8ef34434bf5aafac3c9d0a874 not found: ID does not exist" containerID="d7ae7821917dc2438911ded9942372ca66f915b8ef34434bf5aafac3c9d0a874" Oct 30 13:45:17.456924 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.456864 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ae7821917dc2438911ded9942372ca66f915b8ef34434bf5aafac3c9d0a874"} err="failed to get container status \"d7ae7821917dc2438911ded9942372ca66f915b8ef34434bf5aafac3c9d0a874\": rpc error: code = NotFound desc = could not find container \"d7ae7821917dc2438911ded9942372ca66f915b8ef34434bf5aafac3c9d0a874\": container with ID starting with d7ae7821917dc2438911ded9942372ca66f915b8ef34434bf5aafac3c9d0a874 not found: ID does not exist" Oct 30 13:45:17.488400 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.488364 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1350a65-bf30-427d-be17-0ad9bcf8aa0a" path="/var/lib/kubelet/pods/d1350a65-bf30-427d-be17-0ad9bcf8aa0a/volumes" Oct 30 13:45:17.496895 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.496864 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49"] Oct 30 13:45:17.502940 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:17.502908 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-cda0d9-openshift-default-684c695467-9xp49"] Oct 30 13:45:19.487820 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:19.487787 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" path="/var/lib/kubelet/pods/4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2/volumes" Oct 30 13:45:22.513480 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:22.513429 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p"] Oct 30 13:45:22.514011 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:22.513716 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" podUID="e9284f6a-9945-432a-be97-497fd1a027c5" containerName="istio-proxy" containerID="cri-o://1672ae1231da0275d877b996ee7e2dc86d4704b3d93ea14ce79b651daca7b3de" gracePeriod=30 Oct 30 13:45:27.790259 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.790184 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:45:27.891556 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.891514 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9284f6a-9945-432a-be97-497fd1a027c5-kube-api-access-pr4l7\" (UniqueName: \"kubernetes.io/projected/e9284f6a-9945-432a-be97-497fd1a027c5-kube-api-access-pr4l7\") pod \"e9284f6a-9945-432a-be97-497fd1a027c5\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " Oct 30 13:45:27.891556 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.891560 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-istio-data\") pod \"e9284f6a-9945-432a-be97-497fd1a027c5\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " Oct 30 13:45:27.891817 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.891616 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-workload-socket\") pod \"e9284f6a-9945-432a-be97-497fd1a027c5\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " Oct 30 13:45:27.891817 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.891708 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9284f6a-9945-432a-be97-497fd1a027c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9284f6a-9945-432a-be97-497fd1a027c5-istio-podinfo\") pod \"e9284f6a-9945-432a-be97-497fd1a027c5\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " Oct 30 13:45:27.891817 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.891752 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9284f6a-9945-432a-be97-497fd1a027c5-istio-token\" (UniqueName: \"kubernetes.io/projected/e9284f6a-9945-432a-be97-497fd1a027c5-istio-token\") pod \"e9284f6a-9945-432a-be97-497fd1a027c5\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " Oct 30 13:45:27.891997 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.891838 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-credential-socket\") pod \"e9284f6a-9945-432a-be97-497fd1a027c5\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " Oct 30 13:45:27.891997 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.891889 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9284f6a-9945-432a-be97-497fd1a027c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9284f6a-9945-432a-be97-497fd1a027c5-istiod-ca-cert\") pod \"e9284f6a-9945-432a-be97-497fd1a027c5\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " Oct 30 13:45:27.891997 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.891938 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-istio-data" pod "e9284f6a-9945-432a-be97-497fd1a027c5" (UID: "e9284f6a-9945-432a-be97-497fd1a027c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:27.892161 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.892090 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-workload-socket" pod "e9284f6a-9945-432a-be97-497fd1a027c5" (UID: "e9284f6a-9945-432a-be97-497fd1a027c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:27.892161 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.891928 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-istio-envoy\") pod \"e9284f6a-9945-432a-be97-497fd1a027c5\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " Oct 30 13:45:27.892276 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.892167 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-workload-certs\") pod \"e9284f6a-9945-432a-be97-497fd1a027c5\" (UID: \"e9284f6a-9945-432a-be97-497fd1a027c5\") " Oct 30 13:45:27.892276 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.892187 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-credential-socket" pod "e9284f6a-9945-432a-be97-497fd1a027c5" (UID: "e9284f6a-9945-432a-be97-497fd1a027c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:27.892384 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.892284 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9284f6a-9945-432a-be97-497fd1a027c5-istiod-ca-cert" pod "e9284f6a-9945-432a-be97-497fd1a027c5" (UID: "e9284f6a-9945-432a-be97-497fd1a027c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:27.892506 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.892469 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-workload-certs" pod "e9284f6a-9945-432a-be97-497fd1a027c5" (UID: "e9284f6a-9945-432a-be97-497fd1a027c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:27.892655 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.892627 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:27.892655 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.892655 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9284f6a-9945-432a-be97-497fd1a027c5-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:27.892867 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.892712 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:27.892867 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.892729 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:27.892867 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.892744 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:27.894081 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.894033 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9284f6a-9945-432a-be97-497fd1a027c5-kube-api-access-pr4l7" pod "e9284f6a-9945-432a-be97-497fd1a027c5" (UID: "e9284f6a-9945-432a-be97-497fd1a027c5"). InnerVolumeSpecName "kube-api-access-pr4l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:27.894388 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.894363 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9284f6a-9945-432a-be97-497fd1a027c5-istio-podinfo" pod "e9284f6a-9945-432a-be97-497fd1a027c5" (UID: "e9284f6a-9945-432a-be97-497fd1a027c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:45:27.894537 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.894452 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-istio-envoy" pod "e9284f6a-9945-432a-be97-497fd1a027c5" (UID: "e9284f6a-9945-432a-be97-497fd1a027c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:27.894813 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.894766 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9284f6a-9945-432a-be97-497fd1a027c5-istio-token" pod "e9284f6a-9945-432a-be97-497fd1a027c5" (UID: "e9284f6a-9945-432a-be97-497fd1a027c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:27.993678 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.993638 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9284f6a-9945-432a-be97-497fd1a027c5-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:27.993678 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.993671 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9284f6a-9945-432a-be97-497fd1a027c5-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:27.993678 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.993680 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9284f6a-9945-432a-be97-497fd1a027c5-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:27.993929 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:27.993689 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr4l7\" (UniqueName: \"kubernetes.io/projected/e9284f6a-9945-432a-be97-497fd1a027c5-kube-api-access-pr4l7\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:28.500243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:28.500200 7010 generic.go:356] "Generic (PLEG): container finished" podID="e9284f6a-9945-432a-be97-497fd1a027c5" containerID="1672ae1231da0275d877b996ee7e2dc86d4704b3d93ea14ce79b651daca7b3de" exitCode=0 Oct 30 13:45:28.500243 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:28.500241 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" event={"ID":"e9284f6a-9945-432a-be97-497fd1a027c5","Type":"ContainerDied","Data":"1672ae1231da0275d877b996ee7e2dc86d4704b3d93ea14ce79b651daca7b3de"} Oct 30 13:45:28.500530 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:28.500261 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" Oct 30 13:45:28.500530 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:28.500287 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p" event={"ID":"e9284f6a-9945-432a-be97-497fd1a027c5","Type":"ContainerDied","Data":"cdf1de633425a1781d468ac6e2692272488ce00bb573c50f83afe3a4918cac93"} Oct 30 13:45:28.500530 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:28.500309 7010 scope.go:122] "RemoveContainer" containerID="1672ae1231da0275d877b996ee7e2dc86d4704b3d93ea14ce79b651daca7b3de" Oct 30 13:45:28.510852 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:28.510826 7010 scope.go:122] "RemoveContainer" containerID="1672ae1231da0275d877b996ee7e2dc86d4704b3d93ea14ce79b651daca7b3de" Oct 30 13:45:28.511180 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:28.511159 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1672ae1231da0275d877b996ee7e2dc86d4704b3d93ea14ce79b651daca7b3de\": container with ID starting with 1672ae1231da0275d877b996ee7e2dc86d4704b3d93ea14ce79b651daca7b3de not found: ID does not exist" containerID="1672ae1231da0275d877b996ee7e2dc86d4704b3d93ea14ce79b651daca7b3de" Oct 30 13:45:28.511245 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:28.511191 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1672ae1231da0275d877b996ee7e2dc86d4704b3d93ea14ce79b651daca7b3de"} err="failed to get container status \"1672ae1231da0275d877b996ee7e2dc86d4704b3d93ea14ce79b651daca7b3de\": rpc error: code = NotFound desc = could not find container \"1672ae1231da0275d877b996ee7e2dc86d4704b3d93ea14ce79b651daca7b3de\": container with ID starting with 1672ae1231da0275d877b996ee7e2dc86d4704b3d93ea14ce79b651daca7b3de not found: ID does not exist" Oct 30 13:45:28.568362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:28.568316 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p"] Oct 30 13:45:28.580322 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:28.580285 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f07dfe-openshift-default-86c7789789-6d75p"] Oct 30 13:45:29.400838 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:29.400795 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:45:29.487782 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:29.487741 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9284f6a-9945-432a-be97-497fd1a027c5" path="/var/lib/kubelet/pods/e9284f6a-9945-432a-be97-497fd1a027c5/volumes" Oct 30 13:45:32.064792 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.064751 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" podUID="0895771d-085b-4642-a9ea-3a7549d99831" containerName="main" containerID="cri-o://c87ed39f29a0fb23e601c021b917ac80226faa71829731478055df89825adb3e" gracePeriod=45 Oct 30 13:45:32.080053 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.080016 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" containerID="cri-o://3ead712e4ee52220267b83d1cedfbe7f047a7ae1be6fd45624f6cdefcc885b06" gracePeriod=45 Oct 30 13:45:32.086061 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.086012 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" containerID="cri-o://ee67a97c7963452d1ad4014c4778f9c408b0e71fc4fac5c604a09e10563b0284" gracePeriod=45 Oct 30 13:45:32.107339 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.107294 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" containerID="cri-o://aefc36da9e41ab8814e2e5c444902269bea81da46665b63c48f2bb714d975749" gracePeriod=45 Oct 30 13:45:32.393449 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.393416 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:45:32.421864 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.421836 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:45:32.426657 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.426631 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:45:32.431032 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.431002 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:45:32.522780 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.522745 7010 generic.go:356] "Generic (PLEG): container finished" podID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerID="aefc36da9e41ab8814e2e5c444902269bea81da46665b63c48f2bb714d975749" exitCode=0 Oct 30 13:45:32.522962 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.522804 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" Oct 30 13:45:32.522962 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.522834 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" event={"ID":"ec7b1c88-2656-48b9-90ef-ec21b946341d","Type":"ContainerDied","Data":"aefc36da9e41ab8814e2e5c444902269bea81da46665b63c48f2bb714d975749"} Oct 30 13:45:32.522962 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.522885 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt" event={"ID":"ec7b1c88-2656-48b9-90ef-ec21b946341d","Type":"ContainerDied","Data":"7bca4d32be237ddf03df8b2a999ba296e1957c7316425d2c9a2a699207d945c8"} Oct 30 13:45:32.522962 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.522908 7010 scope.go:122] "RemoveContainer" containerID="aefc36da9e41ab8814e2e5c444902269bea81da46665b63c48f2bb714d975749" Oct 30 13:45:32.525101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.525072 7010 generic.go:356] "Generic (PLEG): container finished" podID="f1940153-057c-44d6-8117-c8746fa9d976" containerID="ee67a97c7963452d1ad4014c4778f9c408b0e71fc4fac5c604a09e10563b0284" exitCode=0 Oct 30 13:45:32.525251 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.525125 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" Oct 30 13:45:32.525251 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.525162 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" event={"ID":"f1940153-057c-44d6-8117-c8746fa9d976","Type":"ContainerDied","Data":"ee67a97c7963452d1ad4014c4778f9c408b0e71fc4fac5c604a09e10563b0284"} Oct 30 13:45:32.525251 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.525207 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp" event={"ID":"f1940153-057c-44d6-8117-c8746fa9d976","Type":"ContainerDied","Data":"aabb19ad753912431b6b29a58abd17f7ff12294e590a8703f00046552779777a"} Oct 30 13:45:32.526827 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.526801 7010 generic.go:356] "Generic (PLEG): container finished" podID="0895771d-085b-4642-a9ea-3a7549d99831" containerID="c87ed39f29a0fb23e601c021b917ac80226faa71829731478055df89825adb3e" exitCode=0 Oct 30 13:45:32.526966 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.526850 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" Oct 30 13:45:32.526966 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.526890 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" event={"ID":"0895771d-085b-4642-a9ea-3a7549d99831","Type":"ContainerDied","Data":"c87ed39f29a0fb23e601c021b917ac80226faa71829731478055df89825adb3e"} Oct 30 13:45:32.526966 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.526932 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" event={"ID":"0895771d-085b-4642-a9ea-3a7549d99831","Type":"ContainerDied","Data":"47428acf6484639c2d5744b40b5c36abc76ab446282dd126112a3e981a12a7a6"} Oct 30 13:45:32.528645 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.528616 7010 generic.go:356] "Generic (PLEG): container finished" podID="0d2f6484-3059-448a-976e-04230dc1f220" containerID="3ead712e4ee52220267b83d1cedfbe7f047a7ae1be6fd45624f6cdefcc885b06" exitCode=0 Oct 30 13:45:32.528779 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.528712 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" Oct 30 13:45:32.528878 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.528861 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" event={"ID":"0d2f6484-3059-448a-976e-04230dc1f220","Type":"ContainerDied","Data":"3ead712e4ee52220267b83d1cedfbe7f047a7ae1be6fd45624f6cdefcc885b06"} Oct 30 13:45:32.528961 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.528906 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2" event={"ID":"0d2f6484-3059-448a-976e-04230dc1f220","Type":"ContainerDied","Data":"6a8fcb097d335f2490fa06b8f0c942677f1f53778fe81f2bc78a035c7334048e"} Oct 30 13:45:32.537863 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.537825 7010 scope.go:122] "RemoveContainer" containerID="aefc36da9e41ab8814e2e5c444902269bea81da46665b63c48f2bb714d975749" Oct 30 13:45:32.538268 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:32.538244 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aefc36da9e41ab8814e2e5c444902269bea81da46665b63c48f2bb714d975749\": container with ID starting with aefc36da9e41ab8814e2e5c444902269bea81da46665b63c48f2bb714d975749 not found: ID does not exist" containerID="aefc36da9e41ab8814e2e5c444902269bea81da46665b63c48f2bb714d975749" Oct 30 13:45:32.538347 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.538280 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aefc36da9e41ab8814e2e5c444902269bea81da46665b63c48f2bb714d975749"} err="failed to get container status \"aefc36da9e41ab8814e2e5c444902269bea81da46665b63c48f2bb714d975749\": rpc error: code = NotFound desc = could not find container \"aefc36da9e41ab8814e2e5c444902269bea81da46665b63c48f2bb714d975749\": container with ID starting with aefc36da9e41ab8814e2e5c444902269bea81da46665b63c48f2bb714d975749 not found: ID does not exist" Oct 30 13:45:32.538347 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.538302 7010 scope.go:122] "RemoveContainer" containerID="ee67a97c7963452d1ad4014c4778f9c408b0e71fc4fac5c604a09e10563b0284" Oct 30 13:45:32.543155 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543119 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-tmp-dir\") pod \"f1940153-057c-44d6-8117-c8746fa9d976\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " Oct 30 13:45:32.543304 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543174 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-tmp-dir\") pod \"0d2f6484-3059-448a-976e-04230dc1f220\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " Oct 30 13:45:32.543304 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543234 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2f6484-3059-448a-976e-04230dc1f220-kube-api-access-62bfs\" (UniqueName: \"kubernetes.io/projected/0d2f6484-3059-448a-976e-04230dc1f220-kube-api-access-62bfs\") pod \"0d2f6484-3059-448a-976e-04230dc1f220\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " Oct 30 13:45:32.543304 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543274 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-tmp-dir\") pod \"ec7b1c88-2656-48b9-90ef-ec21b946341d\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543308 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec7b1c88-2656-48b9-90ef-ec21b946341d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7b1c88-2656-48b9-90ef-ec21b946341d-tls-certs\") pod \"ec7b1c88-2656-48b9-90ef-ec21b946341d\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543333 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-dshm\") pod \"0895771d-085b-4642-a9ea-3a7549d99831\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543364 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-model-cache\") pod \"0895771d-085b-4642-a9ea-3a7549d99831\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543422 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-model-cache\") pod \"f1940153-057c-44d6-8117-c8746fa9d976\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543440 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-tmp-dir" pod "f1940153-057c-44d6-8117-c8746fa9d976" (UID: "f1940153-057c-44d6-8117-c8746fa9d976"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543481 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec7b1c88-2656-48b9-90ef-ec21b946341d-kube-api-access-fq8nl\" (UniqueName: \"kubernetes.io/projected/ec7b1c88-2656-48b9-90ef-ec21b946341d-kube-api-access-fq8nl\") pod \"ec7b1c88-2656-48b9-90ef-ec21b946341d\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543538 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-tmp-dir" pod "0d2f6484-3059-448a-976e-04230dc1f220" (UID: "0d2f6484-3059-448a-976e-04230dc1f220"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543570 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-home\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-home\") pod \"ec7b1c88-2656-48b9-90ef-ec21b946341d\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543595 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-dshm\") pod \"f1940153-057c-44d6-8117-c8746fa9d976\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543598 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-tmp-dir" pod "ec7b1c88-2656-48b9-90ef-ec21b946341d" (UID: "ec7b1c88-2656-48b9-90ef-ec21b946341d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543643 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d2f6484-3059-448a-976e-04230dc1f220-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d2f6484-3059-448a-976e-04230dc1f220-tls-certs\") pod \"0d2f6484-3059-448a-976e-04230dc1f220\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543695 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-home\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-home\") pod \"0d2f6484-3059-448a-976e-04230dc1f220\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543731 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-model-cache\") pod \"0d2f6484-3059-448a-976e-04230dc1f220\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543760 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-model-cache\") pod \"ec7b1c88-2656-48b9-90ef-ec21b946341d\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543762 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-model-cache" pod "f1940153-057c-44d6-8117-c8746fa9d976" (UID: "f1940153-057c-44d6-8117-c8746fa9d976"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543820 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-home" pod "ec7b1c88-2656-48b9-90ef-ec21b946341d" (UID: "ec7b1c88-2656-48b9-90ef-ec21b946341d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543860 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-dshm\") pod \"0d2f6484-3059-448a-976e-04230dc1f220\" (UID: \"0d2f6484-3059-448a-976e-04230dc1f220\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543757 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-model-cache" pod "0895771d-085b-4642-a9ea-3a7549d99831" (UID: "0895771d-085b-4642-a9ea-3a7549d99831"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543899 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-home\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-home\") pod \"0895771d-085b-4642-a9ea-3a7549d99831\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.543970 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0895771d-085b-4642-a9ea-3a7549d99831-tls-certs\" (UniqueName: \"kubernetes.io/secret/0895771d-085b-4642-a9ea-3a7549d99831-tls-certs\") pod \"0895771d-085b-4642-a9ea-3a7549d99831\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.544022 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-dshm\") pod \"ec7b1c88-2656-48b9-90ef-ec21b946341d\" (UID: \"ec7b1c88-2656-48b9-90ef-ec21b946341d\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.544064 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-home\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-home\") pod \"f1940153-057c-44d6-8117-c8746fa9d976\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.544090 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-tmp-dir\") pod \"0895771d-085b-4642-a9ea-3a7549d99831\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.544127 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1940153-057c-44d6-8117-c8746fa9d976-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1940153-057c-44d6-8117-c8746fa9d976-tls-certs\") pod \"f1940153-057c-44d6-8117-c8746fa9d976\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.544174 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0895771d-085b-4642-a9ea-3a7549d99831-kube-api-access-nxx2f\" (UniqueName: \"kubernetes.io/projected/0895771d-085b-4642-a9ea-3a7549d99831-kube-api-access-nxx2f\") pod \"0895771d-085b-4642-a9ea-3a7549d99831\" (UID: \"0895771d-085b-4642-a9ea-3a7549d99831\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.544212 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1940153-057c-44d6-8117-c8746fa9d976-kube-api-access-kdgqk\" (UniqueName: \"kubernetes.io/projected/f1940153-057c-44d6-8117-c8746fa9d976-kube-api-access-kdgqk\") pod \"f1940153-057c-44d6-8117-c8746fa9d976\" (UID: \"f1940153-057c-44d6-8117-c8746fa9d976\") " Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.544670 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.544693 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.544707 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.544722 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.544734 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.544748 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.544007 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-home" pod "0d2f6484-3059-448a-976e-04230dc1f220" (UID: "0d2f6484-3059-448a-976e-04230dc1f220"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.544058 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-home" pod "0895771d-085b-4642-a9ea-3a7549d99831" (UID: "0895771d-085b-4642-a9ea-3a7549d99831"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.544228 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-model-cache" pod "0d2f6484-3059-448a-976e-04230dc1f220" (UID: "0d2f6484-3059-448a-976e-04230dc1f220"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.544398 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-model-cache" pod "ec7b1c88-2656-48b9-90ef-ec21b946341d" (UID: "ec7b1c88-2656-48b9-90ef-ec21b946341d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.544452 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-home" pod "f1940153-057c-44d6-8117-c8746fa9d976" (UID: "f1940153-057c-44d6-8117-c8746fa9d976"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.545282 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.545301 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-tmp-dir" pod "0895771d-085b-4642-a9ea-3a7549d99831" (UID: "0895771d-085b-4642-a9ea-3a7549d99831"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.548806 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.548767 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1940153-057c-44d6-8117-c8746fa9d976-kube-api-access-kdgqk" pod "f1940153-057c-44d6-8117-c8746fa9d976" (UID: "f1940153-057c-44d6-8117-c8746fa9d976"). InnerVolumeSpecName "kube-api-access-kdgqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:32.549026 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.548987 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-dshm" pod "f1940153-057c-44d6-8117-c8746fa9d976" (UID: "f1940153-057c-44d6-8117-c8746fa9d976"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.549597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.549299 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-dshm" pod "0895771d-085b-4642-a9ea-3a7549d99831" (UID: "0895771d-085b-4642-a9ea-3a7549d99831"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.549597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.549353 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-dshm" pod "0d2f6484-3059-448a-976e-04230dc1f220" (UID: "0d2f6484-3059-448a-976e-04230dc1f220"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.549597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.549442 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2f6484-3059-448a-976e-04230dc1f220-kube-api-access-62bfs" pod "0d2f6484-3059-448a-976e-04230dc1f220" (UID: "0d2f6484-3059-448a-976e-04230dc1f220"). InnerVolumeSpecName "kube-api-access-62bfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:32.549597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.549456 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec7b1c88-2656-48b9-90ef-ec21b946341d-tls-certs" pod "ec7b1c88-2656-48b9-90ef-ec21b946341d" (UID: "ec7b1c88-2656-48b9-90ef-ec21b946341d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:32.549597 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.549564 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7b1c88-2656-48b9-90ef-ec21b946341d-kube-api-access-fq8nl" pod "ec7b1c88-2656-48b9-90ef-ec21b946341d" (UID: "ec7b1c88-2656-48b9-90ef-ec21b946341d"). InnerVolumeSpecName "kube-api-access-fq8nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:32.549938 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.549739 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-dshm" pod "ec7b1c88-2656-48b9-90ef-ec21b946341d" (UID: "ec7b1c88-2656-48b9-90ef-ec21b946341d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:32.550145 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.550118 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d2f6484-3059-448a-976e-04230dc1f220-tls-certs" pod "0d2f6484-3059-448a-976e-04230dc1f220" (UID: "0d2f6484-3059-448a-976e-04230dc1f220"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:32.550246 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.550148 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0895771d-085b-4642-a9ea-3a7549d99831-kube-api-access-nxx2f" pod "0895771d-085b-4642-a9ea-3a7549d99831" (UID: "0895771d-085b-4642-a9ea-3a7549d99831"). InnerVolumeSpecName "kube-api-access-nxx2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:32.551565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.551042 7010 scope.go:122] "RemoveContainer" containerID="ee67a97c7963452d1ad4014c4778f9c408b0e71fc4fac5c604a09e10563b0284" Oct 30 13:45:32.551565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.551160 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1940153-057c-44d6-8117-c8746fa9d976-tls-certs" pod "f1940153-057c-44d6-8117-c8746fa9d976" (UID: "f1940153-057c-44d6-8117-c8746fa9d976"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:32.551565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.551338 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0895771d-085b-4642-a9ea-3a7549d99831-tls-certs" pod "0895771d-085b-4642-a9ea-3a7549d99831" (UID: "0895771d-085b-4642-a9ea-3a7549d99831"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:32.551565 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:32.551440 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee67a97c7963452d1ad4014c4778f9c408b0e71fc4fac5c604a09e10563b0284\": container with ID starting with ee67a97c7963452d1ad4014c4778f9c408b0e71fc4fac5c604a09e10563b0284 not found: ID does not exist" containerID="ee67a97c7963452d1ad4014c4778f9c408b0e71fc4fac5c604a09e10563b0284" Oct 30 13:45:32.551950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.551610 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee67a97c7963452d1ad4014c4778f9c408b0e71fc4fac5c604a09e10563b0284"} err="failed to get container status \"ee67a97c7963452d1ad4014c4778f9c408b0e71fc4fac5c604a09e10563b0284\": rpc error: code = NotFound desc = could not find container \"ee67a97c7963452d1ad4014c4778f9c408b0e71fc4fac5c604a09e10563b0284\": container with ID starting with ee67a97c7963452d1ad4014c4778f9c408b0e71fc4fac5c604a09e10563b0284 not found: ID does not exist" Oct 30 13:45:32.551950 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.551650 7010 scope.go:122] "RemoveContainer" containerID="c87ed39f29a0fb23e601c021b917ac80226faa71829731478055df89825adb3e" Oct 30 13:45:32.566568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.563930 7010 scope.go:122] "RemoveContainer" containerID="c87ed39f29a0fb23e601c021b917ac80226faa71829731478055df89825adb3e" Oct 30 13:45:32.566568 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:32.564385 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c87ed39f29a0fb23e601c021b917ac80226faa71829731478055df89825adb3e\": container with ID starting with c87ed39f29a0fb23e601c021b917ac80226faa71829731478055df89825adb3e not found: ID does not exist" containerID="c87ed39f29a0fb23e601c021b917ac80226faa71829731478055df89825adb3e" Oct 30 13:45:32.566568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.564422 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87ed39f29a0fb23e601c021b917ac80226faa71829731478055df89825adb3e"} err="failed to get container status \"c87ed39f29a0fb23e601c021b917ac80226faa71829731478055df89825adb3e\": rpc error: code = NotFound desc = could not find container \"c87ed39f29a0fb23e601c021b917ac80226faa71829731478055df89825adb3e\": container with ID starting with c87ed39f29a0fb23e601c021b917ac80226faa71829731478055df89825adb3e not found: ID does not exist" Oct 30 13:45:32.566568 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.564920 7010 scope.go:122] "RemoveContainer" containerID="3ead712e4ee52220267b83d1cedfbe7f047a7ae1be6fd45624f6cdefcc885b06" Oct 30 13:45:32.576334 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.576288 7010 scope.go:122] "RemoveContainer" containerID="3ead712e4ee52220267b83d1cedfbe7f047a7ae1be6fd45624f6cdefcc885b06" Oct 30 13:45:32.577067 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:32.576719 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ead712e4ee52220267b83d1cedfbe7f047a7ae1be6fd45624f6cdefcc885b06\": container with ID starting with 3ead712e4ee52220267b83d1cedfbe7f047a7ae1be6fd45624f6cdefcc885b06 not found: ID does not exist" containerID="3ead712e4ee52220267b83d1cedfbe7f047a7ae1be6fd45624f6cdefcc885b06" Oct 30 13:45:32.577067 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.576764 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ead712e4ee52220267b83d1cedfbe7f047a7ae1be6fd45624f6cdefcc885b06"} err="failed to get container status \"3ead712e4ee52220267b83d1cedfbe7f047a7ae1be6fd45624f6cdefcc885b06\": rpc error: code = NotFound desc = could not find container \"3ead712e4ee52220267b83d1cedfbe7f047a7ae1be6fd45624f6cdefcc885b06\": container with ID starting with 3ead712e4ee52220267b83d1cedfbe7f047a7ae1be6fd45624f6cdefcc885b06 not found: ID does not exist" Oct 30 13:45:32.645881 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.645842 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.645881 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.645875 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.645881 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.645891 7010 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-tmp-dir\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.646205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.645904 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1940153-057c-44d6-8117-c8746fa9d976-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.646205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.645918 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxx2f\" (UniqueName: \"kubernetes.io/projected/0895771d-085b-4642-a9ea-3a7549d99831-kube-api-access-nxx2f\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.646205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.645932 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdgqk\" (UniqueName: \"kubernetes.io/projected/f1940153-057c-44d6-8117-c8746fa9d976-kube-api-access-kdgqk\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.646205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.645948 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62bfs\" (UniqueName: \"kubernetes.io/projected/0d2f6484-3059-448a-976e-04230dc1f220-kube-api-access-62bfs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.646205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.645967 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7b1c88-2656-48b9-90ef-ec21b946341d-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.646205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.645980 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.646205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.645994 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq8nl\" (UniqueName: \"kubernetes.io/projected/ec7b1c88-2656-48b9-90ef-ec21b946341d-kube-api-access-fq8nl\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.646205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.646006 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1940153-057c-44d6-8117-c8746fa9d976-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.646205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.646020 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d2f6484-3059-448a-976e-04230dc1f220-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.646205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.646030 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.646205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.646038 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.646205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.646047 7010 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7b1c88-2656-48b9-90ef-ec21b946341d-model-cache\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.646205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.646054 7010 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d2f6484-3059-448a-976e-04230dc1f220-dshm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.646205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.646062 7010 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0895771d-085b-4642-a9ea-3a7549d99831-home\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.646205 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.646069 7010 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0895771d-085b-4642-a9ea-3a7549d99831-tls-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:32.875457 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.875417 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt"] Oct 30 13:45:32.880549 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.880513 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/simulator-w6-ee7d65-kserve-59d49dcd9-9z6dt"] Oct 30 13:45:32.915946 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.915848 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2"] Oct 30 13:45:32.920938 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.920904 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/distinct-w6-ee7d65-kserve-7c75b64879-5pxr2"] Oct 30 13:45:32.958565 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.958524 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp"] Oct 30 13:45:32.962458 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.962405 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/premium-w6-ee7d65-kserve-97b8977fb-fr8lp"] Oct 30 13:45:32.997540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:32.997500 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt"] Oct 30 13:45:33.010177 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:33.010145 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt"] Oct 30 13:45:33.308241 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:33.308143 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ee7d65/distinct-2-w6-ee7d65-kserve-7bdd787c44-cxkrt" podUID="0895771d-085b-4642-a9ea-3a7549d99831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 13:45:33.486949 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:33.486912 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0895771d-085b-4642-a9ea-3a7549d99831" path="/var/lib/kubelet/pods/0895771d-085b-4642-a9ea-3a7549d99831/volumes" Oct 30 13:45:33.487337 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:33.487323 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2f6484-3059-448a-976e-04230dc1f220" path="/var/lib/kubelet/pods/0d2f6484-3059-448a-976e-04230dc1f220/volumes" Oct 30 13:45:33.487717 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:33.487703 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" path="/var/lib/kubelet/pods/ec7b1c88-2656-48b9-90ef-ec21b946341d/volumes" Oct 30 13:45:33.488395 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:33.488379 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1940153-057c-44d6-8117-c8746fa9d976" path="/var/lib/kubelet/pods/f1940153-057c-44d6-8117-c8746fa9d976/volumes" Oct 30 13:45:40.270034 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.269996 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf"] Oct 30 13:45:40.270676 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.270300 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" podUID="e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8" containerName="maas-api" containerID="cri-o://43b92a7406a2ca3115664a6077cfe3c66cc94979dbe6d75b1a8ea9431f2ce4ea" gracePeriod=30 Oct 30 13:45:40.346194 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.346130 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" podUID="e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.32:8443/health\": dial tcp 10.134.0.32:8443: connect: connection refused" Oct 30 13:45:40.566637 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.566608 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:45:40.577475 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.577435 7010 generic.go:356] "Generic (PLEG): container finished" podID="e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8" containerID="43b92a7406a2ca3115664a6077cfe3c66cc94979dbe6d75b1a8ea9431f2ce4ea" exitCode=0 Oct 30 13:45:40.577637 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.577482 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" Oct 30 13:45:40.577637 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.577526 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" event={"ID":"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8","Type":"ContainerDied","Data":"43b92a7406a2ca3115664a6077cfe3c66cc94979dbe6d75b1a8ea9431f2ce4ea"} Oct 30 13:45:40.577637 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.577581 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf" event={"ID":"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8","Type":"ContainerDied","Data":"937f2a8fb52696d059be44b29bc83359a7e98fe8f59ad4206c878d6ad596e980"} Oct 30 13:45:40.577637 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.577606 7010 scope.go:122] "RemoveContainer" containerID="43b92a7406a2ca3115664a6077cfe3c66cc94979dbe6d75b1a8ea9431f2ce4ea" Oct 30 13:45:40.587580 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.587554 7010 scope.go:122] "RemoveContainer" containerID="43b92a7406a2ca3115664a6077cfe3c66cc94979dbe6d75b1a8ea9431f2ce4ea" Oct 30 13:45:40.587978 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:40.587951 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b92a7406a2ca3115664a6077cfe3c66cc94979dbe6d75b1a8ea9431f2ce4ea\": container with ID starting with 43b92a7406a2ca3115664a6077cfe3c66cc94979dbe6d75b1a8ea9431f2ce4ea not found: ID does not exist" containerID="43b92a7406a2ca3115664a6077cfe3c66cc94979dbe6d75b1a8ea9431f2ce4ea" Oct 30 13:45:40.588086 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.587991 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b92a7406a2ca3115664a6077cfe3c66cc94979dbe6d75b1a8ea9431f2ce4ea"} err="failed to get container status \"43b92a7406a2ca3115664a6077cfe3c66cc94979dbe6d75b1a8ea9431f2ce4ea\": rpc error: code = NotFound desc = could not find container \"43b92a7406a2ca3115664a6077cfe3c66cc94979dbe6d75b1a8ea9431f2ce4ea\": container with ID starting with 43b92a7406a2ca3115664a6077cfe3c66cc94979dbe6d75b1a8ea9431f2ce4ea not found: ID does not exist" Oct 30 13:45:40.619038 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.619002 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-kube-api-access-wdggr\" (UniqueName: \"kubernetes.io/projected/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-kube-api-access-wdggr\") pod \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\" (UID: \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\") " Oct 30 13:45:40.619244 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.619063 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-maas-api-tls\") pod \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\" (UID: \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\") " Oct 30 13:45:40.619244 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.619115 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-metrics-certs\") pod \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\" (UID: \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\") " Oct 30 13:45:40.619244 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.619219 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-openshift-service-ca\") pod \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\" (UID: \"e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8\") " Oct 30 13:45:40.620481 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.620437 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-openshift-service-ca" pod "e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8" (UID: "e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:40.623150 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.623077 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-kube-api-access-wdggr" pod "e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8" (UID: "e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8"). InnerVolumeSpecName "kube-api-access-wdggr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:40.623150 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.623079 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-metrics-certs" pod "e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8" (UID: "e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:40.633331 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.633282 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-maas-api-tls" pod "e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8" (UID: "e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:40.721107 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.720997 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:40.721107 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.721039 7010 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-openshift-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:40.721107 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.721058 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdggr\" (UniqueName: \"kubernetes.io/projected/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-kube-api-access-wdggr\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:40.721107 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.721077 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8-maas-api-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:40.934600 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.934557 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf"] Oct 30 13:45:40.946813 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:40.946776 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ee7d65-75bcf9cdb6-qrmtf"] Oct 30 13:45:41.491805 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:41.491762 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8" path="/var/lib/kubelet/pods/e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8/volumes" Oct 30 13:45:48.430427 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:48.430394 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv"] Oct 30 13:45:48.430902 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:48.430684 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" podUID="48852ae1-935d-4cfc-acd4-e2082a6ad5ab" containerName="istio-proxy" containerID="cri-o://52bc9fda63c1c215fdd284723c6c152c056cd5ef9734269f4f3b9d17c3b49df5" gracePeriod=30 Oct 30 13:45:53.430543 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.430483 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b844f68b7-bf8nh"] Oct 30 13:45:53.431023 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.430783 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-b844f68b7-bf8nh" podUID="67b12db1-48f1-4058-94f9-ce7bb7098add" containerName="manager" containerID="cri-o://b48b4dc68c92bb340db7015e629a69f3a42d8a2a3e7a9b547c22e9f6c39b2f33" gracePeriod=30 Oct 30 13:45:53.656826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.656687 7010 generic.go:356] "Generic (PLEG): container finished" podID="48852ae1-935d-4cfc-acd4-e2082a6ad5ab" containerID="52bc9fda63c1c215fdd284723c6c152c056cd5ef9734269f4f3b9d17c3b49df5" exitCode=0 Oct 30 13:45:53.656826 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.656761 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" event={"ID":"48852ae1-935d-4cfc-acd4-e2082a6ad5ab","Type":"ContainerDied","Data":"52bc9fda63c1c215fdd284723c6c152c056cd5ef9734269f4f3b9d17c3b49df5"} Oct 30 13:45:53.658579 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.658547 7010 generic.go:356] "Generic (PLEG): container finished" podID="67b12db1-48f1-4058-94f9-ce7bb7098add" containerID="b48b4dc68c92bb340db7015e629a69f3a42d8a2a3e7a9b547c22e9f6c39b2f33" exitCode=0 Oct 30 13:45:53.658750 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.658600 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b844f68b7-bf8nh" event={"ID":"67b12db1-48f1-4058-94f9-ce7bb7098add","Type":"ContainerDied","Data":"b48b4dc68c92bb340db7015e629a69f3a42d8a2a3e7a9b547c22e9f6c39b2f33"} Oct 30 13:45:53.721737 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.721693 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b844f68b7-bf8nh" Oct 30 13:45:53.765427 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.765378 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:45:53.853832 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.853793 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-envoy\") pod \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " Oct 30 13:45:53.853832 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.853836 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-podinfo\") pod \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " Oct 30 13:45:53.854110 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.853861 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67b12db1-48f1-4058-94f9-ce7bb7098add-metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b12db1-48f1-4058-94f9-ce7bb7098add-metrics-certs\") pod \"67b12db1-48f1-4058-94f9-ce7bb7098add\" (UID: \"67b12db1-48f1-4058-94f9-ce7bb7098add\") " Oct 30 13:45:53.854110 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.853906 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-token\" (UniqueName: \"kubernetes.io/projected/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-token\") pod \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " Oct 30 13:45:53.854110 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.853967 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b12db1-48f1-4058-94f9-ce7bb7098add-kube-api-access-rhzph\" (UniqueName: \"kubernetes.io/projected/67b12db1-48f1-4058-94f9-ce7bb7098add-kube-api-access-rhzph\") pod \"67b12db1-48f1-4058-94f9-ce7bb7098add\" (UID: \"67b12db1-48f1-4058-94f9-ce7bb7098add\") " Oct 30 13:45:53.854404 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.854377 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67b12db1-48f1-4058-94f9-ce7bb7098add-webhook-cert\" (UniqueName: \"kubernetes.io/secret/67b12db1-48f1-4058-94f9-ce7bb7098add-webhook-cert\") pod \"67b12db1-48f1-4058-94f9-ce7bb7098add\" (UID: \"67b12db1-48f1-4058-94f9-ce7bb7098add\") " Oct 30 13:45:53.854578 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.854455 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-credential-socket\") pod \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " Oct 30 13:45:53.854578 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.854510 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-workload-socket\") pod \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " Oct 30 13:45:53.854578 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.854546 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-workload-certs\") pod \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " Oct 30 13:45:53.854787 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.854605 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-kube-api-access-q4trs\" (UniqueName: \"kubernetes.io/projected/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-kube-api-access-q4trs\") pod \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " Oct 30 13:45:53.854787 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.854642 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istiod-ca-cert\") pod \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " Oct 30 13:45:53.854787 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.854706 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-data\") pod \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\" (UID: \"48852ae1-935d-4cfc-acd4-e2082a6ad5ab\") " Oct 30 13:45:53.855663 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.855298 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-workload-certs" pod "48852ae1-935d-4cfc-acd4-e2082a6ad5ab" (UID: "48852ae1-935d-4cfc-acd4-e2082a6ad5ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:53.855663 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.855327 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-workload-socket" pod "48852ae1-935d-4cfc-acd4-e2082a6ad5ab" (UID: "48852ae1-935d-4cfc-acd4-e2082a6ad5ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:53.855663 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.855338 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-credential-socket" pod "48852ae1-935d-4cfc-acd4-e2082a6ad5ab" (UID: "48852ae1-935d-4cfc-acd4-e2082a6ad5ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:53.855663 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.855572 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-data" pod "48852ae1-935d-4cfc-acd4-e2082a6ad5ab" (UID: "48852ae1-935d-4cfc-acd4-e2082a6ad5ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:53.855962 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.855659 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istiod-ca-cert" pod "48852ae1-935d-4cfc-acd4-e2082a6ad5ab" (UID: "48852ae1-935d-4cfc-acd4-e2082a6ad5ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:45:53.857375 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.857345 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-token" pod "48852ae1-935d-4cfc-acd4-e2082a6ad5ab" (UID: "48852ae1-935d-4cfc-acd4-e2082a6ad5ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:53.857535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.857395 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-envoy" pod "48852ae1-935d-4cfc-acd4-e2082a6ad5ab" (UID: "48852ae1-935d-4cfc-acd4-e2082a6ad5ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:45:53.857535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.857421 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-podinfo" pod "48852ae1-935d-4cfc-acd4-e2082a6ad5ab" (UID: "48852ae1-935d-4cfc-acd4-e2082a6ad5ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:45:53.857535 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.857432 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b12db1-48f1-4058-94f9-ce7bb7098add-metrics-certs" pod "67b12db1-48f1-4058-94f9-ce7bb7098add" (UID: "67b12db1-48f1-4058-94f9-ce7bb7098add"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:53.857842 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.857815 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-kube-api-access-q4trs" pod "48852ae1-935d-4cfc-acd4-e2082a6ad5ab" (UID: "48852ae1-935d-4cfc-acd4-e2082a6ad5ab"). InnerVolumeSpecName "kube-api-access-q4trs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:53.858206 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.858180 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b12db1-48f1-4058-94f9-ce7bb7098add-kube-api-access-rhzph" pod "67b12db1-48f1-4058-94f9-ce7bb7098add" (UID: "67b12db1-48f1-4058-94f9-ce7bb7098add"). InnerVolumeSpecName "kube-api-access-rhzph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:45:53.858693 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.858670 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b12db1-48f1-4058-94f9-ce7bb7098add-webhook-cert" pod "67b12db1-48f1-4058-94f9-ce7bb7098add" (UID: "67b12db1-48f1-4058-94f9-ce7bb7098add"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:45:53.956065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.955968 7010 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-envoy\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:53.956065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.956004 7010 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-podinfo\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:53.956065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.956016 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b12db1-48f1-4058-94f9-ce7bb7098add-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:53.956065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.956028 7010 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-token\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:53.956065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.956040 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhzph\" (UniqueName: \"kubernetes.io/projected/67b12db1-48f1-4058-94f9-ce7bb7098add-kube-api-access-rhzph\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:53.956065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.956053 7010 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67b12db1-48f1-4058-94f9-ce7bb7098add-webhook-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:53.956065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.956066 7010 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-credential-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:53.956445 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.956077 7010 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-workload-socket\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:53.956445 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.956092 7010 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-workload-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:53.956445 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.956103 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4trs\" (UniqueName: \"kubernetes.io/projected/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-kube-api-access-q4trs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:53.956445 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.956116 7010 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istiod-ca-cert\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:53.956445 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:53.956128 7010 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48852ae1-935d-4cfc-acd4-e2082a6ad5ab-istio-data\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:45:54.664923 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:54.664889 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b844f68b7-bf8nh" Oct 30 13:45:54.665410 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:54.664926 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b844f68b7-bf8nh" event={"ID":"67b12db1-48f1-4058-94f9-ce7bb7098add","Type":"ContainerDied","Data":"c4e342cb4b576554e1f08bf4b83548dde0509655841845a5fe54cf2468e509e1"} Oct 30 13:45:54.665410 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:54.664985 7010 scope.go:122] "RemoveContainer" containerID="b48b4dc68c92bb340db7015e629a69f3a42d8a2a3e7a9b547c22e9f6c39b2f33" Oct 30 13:45:54.666440 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:54.666413 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" event={"ID":"48852ae1-935d-4cfc-acd4-e2082a6ad5ab","Type":"ContainerDied","Data":"8af16105b8d8e68bd47cd50ae060cabe2018eba306454330ac4ef8c611d3bfdd"} Oct 30 13:45:54.666564 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:54.666446 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv" Oct 30 13:45:54.677665 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:54.677647 7010 scope.go:122] "RemoveContainer" containerID="52bc9fda63c1c215fdd284723c6c152c056cd5ef9734269f4f3b9d17c3b49df5" Oct 30 13:45:54.791662 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:54.791572 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv"] Oct 30 13:45:54.804195 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:54.804162 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee7d65-openshift-default-7d844fb5d9-n4xmv"] Oct 30 13:45:54.866685 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:54.866647 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b844f68b7-bf8nh"] Oct 30 13:45:54.870526 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:54.870470 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b844f68b7-bf8nh"] Oct 30 13:45:55.488393 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:55.488359 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48852ae1-935d-4cfc-acd4-e2082a6ad5ab" path="/var/lib/kubelet/pods/48852ae1-935d-4cfc-acd4-e2082a6ad5ab/volumes" Oct 30 13:45:55.488778 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:45:55.488764 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b12db1-48f1-4058-94f9-ce7bb7098add" path="/var/lib/kubelet/pods/67b12db1-48f1-4058-94f9-ce7bb7098add/volumes" Oct 30 13:45:59.399664 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:45:59.399627 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:46:25.409946 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.409905 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6"] Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410302 7010 cpu_manager.go:414] "containerMap: removing container" podUID="2d8103d2-d203-4fc6-9e85-e071908539ab" containerName="istio-proxy" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410315 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8103d2-d203-4fc6-9e85-e071908539ab" containerName="istio-proxy" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410333 7010 cpu_manager.go:414] "containerMap: removing container" podUID="e9284f6a-9945-432a-be97-497fd1a027c5" containerName="istio-proxy" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410338 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9284f6a-9945-432a-be97-497fd1a027c5" containerName="istio-proxy" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410346 7010 cpu_manager.go:414] "containerMap: removing container" podUID="67b12db1-48f1-4058-94f9-ce7bb7098add" containerName="manager" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410351 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b12db1-48f1-4058-94f9-ce7bb7098add" containerName="manager" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410363 7010 cpu_manager.go:414] "containerMap: removing container" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410368 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410376 7010 cpu_manager.go:414] "containerMap: removing container" podUID="4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" containerName="istio-proxy" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410381 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" containerName="istio-proxy" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410388 7010 cpu_manager.go:414] "containerMap: removing container" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410393 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410399 7010 cpu_manager.go:414] "containerMap: removing container" podUID="3575f061-f8d0-4915-b4e8-c88c882407e5" containerName="istio-proxy" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410403 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3575f061-f8d0-4915-b4e8-c88c882407e5" containerName="istio-proxy" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410408 7010 cpu_manager.go:414] "containerMap: removing container" podUID="37fbf9d2-f376-4f7d-a514-54e51342f2f6" containerName="maas-api" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410413 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37fbf9d2-f376-4f7d-a514-54e51342f2f6" containerName="maas-api" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410423 7010 cpu_manager.go:414] "containerMap: removing container" podUID="36015964-d36a-428a-8c47-367689174d12" containerName="istio-proxy" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410427 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36015964-d36a-428a-8c47-367689174d12" containerName="istio-proxy" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410437 7010 cpu_manager.go:414] "containerMap: removing container" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410441 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410448 7010 cpu_manager.go:414] "containerMap: removing container" podUID="4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" containerName="istio-proxy" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410452 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" containerName="istio-proxy" Oct 30 13:46:25.410448 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410459 7010 cpu_manager.go:414] "containerMap: removing container" podUID="7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410464 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410471 7010 cpu_manager.go:414] "containerMap: removing container" podUID="48852ae1-935d-4cfc-acd4-e2082a6ad5ab" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410475 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48852ae1-935d-4cfc-acd4-e2082a6ad5ab" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410481 7010 cpu_manager.go:414] "containerMap: removing container" podUID="4e411b13-5f67-4954-bdcd-582026eb0ac4" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410499 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e411b13-5f67-4954-bdcd-582026eb0ac4" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410507 7010 cpu_manager.go:414] "containerMap: removing container" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410511 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410518 7010 cpu_manager.go:414] "containerMap: removing container" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410522 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410528 7010 cpu_manager.go:414] "containerMap: removing container" podUID="246d0d04-97d3-4762-9075-777053be1de9" containerName="payload-pre-processing" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410532 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="246d0d04-97d3-4762-9075-777053be1de9" containerName="payload-pre-processing" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410542 7010 cpu_manager.go:414] "containerMap: removing container" podUID="0895771d-085b-4642-a9ea-3a7549d99831" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410547 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0895771d-085b-4642-a9ea-3a7549d99831" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410553 7010 cpu_manager.go:414] "containerMap: removing container" podUID="a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410557 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410563 7010 cpu_manager.go:414] "containerMap: removing container" podUID="d1350a65-bf30-427d-be17-0ad9bcf8aa0a" containerName="maas-api" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410567 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1350a65-bf30-427d-be17-0ad9bcf8aa0a" containerName="maas-api" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410574 7010 cpu_manager.go:414] "containerMap: removing container" podUID="3575f061-f8d0-4915-b4e8-c88c882407e5" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410578 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3575f061-f8d0-4915-b4e8-c88c882407e5" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410589 7010 cpu_manager.go:414] "containerMap: removing container" podUID="fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410593 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd9b0d7-0eb8-4a7b-b668-045d7ff7c231" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410599 7010 cpu_manager.go:414] "containerMap: removing container" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410603 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410609 7010 cpu_manager.go:414] "containerMap: removing container" podUID="e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8" containerName="maas-api" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410613 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8" containerName="maas-api" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410622 7010 cpu_manager.go:414] "containerMap: removing container" podUID="78b88187-8364-42ca-84dd-9e3d06c1ee3b" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410627 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b88187-8364-42ca-84dd-9e3d06c1ee3b" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410677 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="48852ae1-935d-4cfc-acd4-e2082a6ad5ab" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410686 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e411b13-5f67-4954-bdcd-582026eb0ac4" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410693 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1940153-057c-44d6-8117-c8746fa9d976" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410700 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2f6484-3059-448a-976e-04230dc1f220" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410706 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="246d0d04-97d3-4762-9075-777053be1de9" containerName="payload-pre-processing" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410712 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="0895771d-085b-4642-a9ea-3a7549d99831" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410718 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d8d8e3-f2ca-45e5-bd85-7454ddac6d53" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410724 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1350a65-bf30-427d-be17-0ad9bcf8aa0a" containerName="maas-api" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410730 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="3575f061-f8d0-4915-b4e8-c88c882407e5" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410736 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ceac9f-ae34-46f7-b04d-7d1484f880d7" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410742 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1c6fc6e-3eea-4135-85e2-e68b79d9b5c8" containerName="maas-api" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410747 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b88187-8364-42ca-84dd-9e3d06c1ee3b" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410752 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8103d2-d203-4fc6-9e85-e071908539ab" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410757 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9284f6a-9945-432a-be97-497fd1a027c5" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410762 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b12db1-48f1-4058-94f9-ce7bb7098add" containerName="manager" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410767 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="7393071a-3009-43b4-afad-0ba041babef4" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410772 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410777 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7b1c88-2656-48b9-90ef-ec21b946341d" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410782 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="37fbf9d2-f376-4f7d-a514-54e51342f2f6" containerName="maas-api" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410788 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="36015964-d36a-428a-8c47-367689174d12" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410792 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56893d6-cf58-4ba8-b5b1-a5dc7780d7e9" containerName="main" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410798 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ebd68ad-3bc6-4c67-8f3f-92ac77eb1ba2" containerName="istio-proxy" Oct 30 13:46:25.411180 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.410802 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ffbc852-329e-4a5a-a9ea-0cba3a63ec6e" containerName="istio-proxy" Oct 30 13:46:25.413780 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.413758 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.437049 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.437011 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6"] Oct 30 13:46:25.450883 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.450845 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c7855fd-e6a6-44ad-870e-ed9f483528f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.451065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.450899 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c7855fd-e6a6-44ad-870e-ed9f483528f2-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.451065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.450944 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c7855fd-e6a6-44ad-870e-ed9f483528f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.451065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.450982 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c7855fd-e6a6-44ad-870e-ed9f483528f2-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.451065 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.451024 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c7855fd-e6a6-44ad-870e-ed9f483528f2-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.451250 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.451173 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c7855fd-e6a6-44ad-870e-ed9f483528f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.451250 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.451204 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c7855fd-e6a6-44ad-870e-ed9f483528f2-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.451250 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.451225 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmrh2\" (UniqueName: \"kubernetes.io/projected/3c7855fd-e6a6-44ad-870e-ed9f483528f2-kube-api-access-xmrh2\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.451362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.451251 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c7855fd-e6a6-44ad-870e-ed9f483528f2-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.551930 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.551893 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c7855fd-e6a6-44ad-870e-ed9f483528f2-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.552101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.551980 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c7855fd-e6a6-44ad-870e-ed9f483528f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.552101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.552005 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c7855fd-e6a6-44ad-870e-ed9f483528f2-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.552101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.552034 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrh2\" (UniqueName: \"kubernetes.io/projected/3c7855fd-e6a6-44ad-870e-ed9f483528f2-kube-api-access-xmrh2\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.552101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.552079 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c7855fd-e6a6-44ad-870e-ed9f483528f2-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.552296 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.552117 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c7855fd-e6a6-44ad-870e-ed9f483528f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.552296 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.552159 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c7855fd-e6a6-44ad-870e-ed9f483528f2-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.552296 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.552184 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c7855fd-e6a6-44ad-870e-ed9f483528f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.552296 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.552209 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c7855fd-e6a6-44ad-870e-ed9f483528f2-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.552551 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.552383 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c7855fd-e6a6-44ad-870e-ed9f483528f2-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.552616 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.552575 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c7855fd-e6a6-44ad-870e-ed9f483528f2-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.552743 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.552721 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c7855fd-e6a6-44ad-870e-ed9f483528f2-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.552803 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.552729 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c7855fd-e6a6-44ad-870e-ed9f483528f2-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.553012 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.552992 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c7855fd-e6a6-44ad-870e-ed9f483528f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.554540 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.554516 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c7855fd-e6a6-44ad-870e-ed9f483528f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.554739 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.554720 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c7855fd-e6a6-44ad-870e-ed9f483528f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.577080 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.577051 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c7855fd-e6a6-44ad-870e-ed9f483528f2-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.577359 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.577336 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmrh2\" (UniqueName: \"kubernetes.io/projected/3c7855fd-e6a6-44ad-870e-ed9f483528f2-kube-api-access-xmrh2\") pod \"maas-default-gateway-openshift-default-64d4cc998c-tlqh6\" (UID: \"3c7855fd-e6a6-44ad-870e-ed9f483528f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:25.728628 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:25.728526 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:26.051947 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:26.051920 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6"] Oct 30 13:46:26.057864 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:46:26.056772 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7855fd_e6a6_44ad_870e_ed9f483528f2.slice/crio-10d9d6c4f536ea2fc5131efa0db5202c6eecc0fc3dc097ab30b454f2142e1296 WatchSource:0}: Error finding container 10d9d6c4f536ea2fc5131efa0db5202c6eecc0fc3dc097ab30b454f2142e1296: Status 404 returned error can't find the container with id 10d9d6c4f536ea2fc5131efa0db5202c6eecc0fc3dc097ab30b454f2142e1296 Oct 30 13:46:26.059143 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:26.059100 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:46:26.059233 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:26.059190 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:46:26.059233 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:26.059228 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:46:26.821803 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:26.821764 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" event={"ID":"3c7855fd-e6a6-44ad-870e-ed9f483528f2","Type":"ContainerStarted","Data":"a4b15c1ea22d3ec4d845fef50a4a26f9f6d46b065f78ebc4e4ab79cd6476c6d2"} Oct 30 13:46:26.822234 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:26.821826 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" event={"ID":"3c7855fd-e6a6-44ad-870e-ed9f483528f2","Type":"ContainerStarted","Data":"10d9d6c4f536ea2fc5131efa0db5202c6eecc0fc3dc097ab30b454f2142e1296"} Oct 30 13:46:26.883263 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:26.883208 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" podStartSLOduration=1.883194236 podStartE2EDuration="1.883194236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:46:26.882395597 +0000 UTC m=+3243.514582761" watchObservedRunningTime="2026-10-30 13:46:26.883194236 +0000 UTC m=+3243.515381398" Oct 30 13:46:27.728992 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:27.728952 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:27.730440 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:27.730415 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-tlqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 13:46:27.730523 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:27.730466 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" podUID="3c7855fd-e6a6-44ad-870e-ed9f483528f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 13:46:28.729216 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:28.729171 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-tlqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 13:46:28.729666 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:28.729235 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" podUID="3c7855fd-e6a6-44ad-870e-ed9f483528f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 13:46:29.421541 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:46:29.421481 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:46:29.729625 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:29.729545 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-tlqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 13:46:29.729625 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:29.729609 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" podUID="3c7855fd-e6a6-44ad-870e-ed9f483528f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 13:46:30.729173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:30.729138 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-tlqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 13:46:30.729371 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:30.729193 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" podUID="3c7855fd-e6a6-44ad-870e-ed9f483528f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 13:46:31.729126 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:31.729091 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-tlqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 13:46:31.729570 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:31.729147 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" podUID="3c7855fd-e6a6-44ad-870e-ed9f483528f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 13:46:32.729846 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:32.729801 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-tlqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 13:46:32.730265 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:32.729871 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" podUID="3c7855fd-e6a6-44ad-870e-ed9f483528f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 13:46:34.314742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:34.314661 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:34.315196 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:34.315044 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:34.315848 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:46:34.315827 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-tlqh6" Oct 30 13:46:59.398640 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:46:59.398600 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:47:24.049889 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:24.049862 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:47:24.072193 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:24.072168 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:47:25.649125 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.649086 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s"] Oct 30 13:47:25.649582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.649567 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="3575f061-f8d0-4915-b4e8-c88c882407e5" containerName="istio-proxy" Oct 30 13:47:25.652538 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.652517 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.692432 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.692404 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s"] Oct 30 13:47:25.841923 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.841888 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/712fe9d3-fad8-4615-b9ba-18eeee7ec299-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.842094 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.841929 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/712fe9d3-fad8-4615-b9ba-18eeee7ec299-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.842094 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.841961 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/712fe9d3-fad8-4615-b9ba-18eeee7ec299-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.842094 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.842033 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/712fe9d3-fad8-4615-b9ba-18eeee7ec299-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.842207 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.842130 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/712fe9d3-fad8-4615-b9ba-18eeee7ec299-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.842207 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.842190 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwznb\" (UniqueName: \"kubernetes.io/projected/712fe9d3-fad8-4615-b9ba-18eeee7ec299-kube-api-access-xwznb\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.842273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.842236 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/712fe9d3-fad8-4615-b9ba-18eeee7ec299-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.842273 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.842260 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/712fe9d3-fad8-4615-b9ba-18eeee7ec299-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.842388 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.842295 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/712fe9d3-fad8-4615-b9ba-18eeee7ec299-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.943004 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.942924 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/712fe9d3-fad8-4615-b9ba-18eeee7ec299-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.943004 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.942970 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwznb\" (UniqueName: \"kubernetes.io/projected/712fe9d3-fad8-4615-b9ba-18eeee7ec299-kube-api-access-xwznb\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.943004 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.943002 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/712fe9d3-fad8-4615-b9ba-18eeee7ec299-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.943348 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.943025 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/712fe9d3-fad8-4615-b9ba-18eeee7ec299-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.943348 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.943050 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/712fe9d3-fad8-4615-b9ba-18eeee7ec299-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.943348 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.943079 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/712fe9d3-fad8-4615-b9ba-18eeee7ec299-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.943348 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.943108 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/712fe9d3-fad8-4615-b9ba-18eeee7ec299-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.943348 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.943220 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/712fe9d3-fad8-4615-b9ba-18eeee7ec299-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.943348 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.943254 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/712fe9d3-fad8-4615-b9ba-18eeee7ec299-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.943731 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.943364 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/712fe9d3-fad8-4615-b9ba-18eeee7ec299-workload-certs\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.943731 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.943465 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/712fe9d3-fad8-4615-b9ba-18eeee7ec299-credential-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.943731 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.943513 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/712fe9d3-fad8-4615-b9ba-18eeee7ec299-workload-socket\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.943731 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.943642 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/712fe9d3-fad8-4615-b9ba-18eeee7ec299-istio-data\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.943894 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.943875 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/712fe9d3-fad8-4615-b9ba-18eeee7ec299-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.945585 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.945566 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/712fe9d3-fad8-4615-b9ba-18eeee7ec299-istio-envoy\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.945805 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.945789 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/712fe9d3-fad8-4615-b9ba-18eeee7ec299-istio-podinfo\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.977802 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.977774 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwznb\" (UniqueName: \"kubernetes.io/projected/712fe9d3-fad8-4615-b9ba-18eeee7ec299-kube-api-access-xwznb\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:25.977949 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:25.977771 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/712fe9d3-fad8-4615-b9ba-18eeee7ec299-istio-token\") pod \"maas-default-gateway-openshift-default-64d4cc998c-sm24s\" (UID: \"712fe9d3-fad8-4615-b9ba-18eeee7ec299\") " pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:26.266908 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:26.266817 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:26.598357 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:26.597987 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s"] Oct 30 13:47:26.602193 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:47:26.602162 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712fe9d3_fad8_4615_b9ba_18eeee7ec299.slice/crio-c0a3412a5ecaba1b566574315a5218a29133020f7ccbe79bfecce1826b31c552 WatchSource:0}: Error finding container c0a3412a5ecaba1b566574315a5218a29133020f7ccbe79bfecce1826b31c552: Status 404 returned error can't find the container with id c0a3412a5ecaba1b566574315a5218a29133020f7ccbe79bfecce1826b31c552 Oct 30 13:47:26.605187 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:26.605140 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:47:26.605304 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:26.605237 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:47:26.605304 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:26.605285 7010 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 13:47:27.098296 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:27.098253 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" event={"ID":"712fe9d3-fad8-4615-b9ba-18eeee7ec299","Type":"ContainerStarted","Data":"e169b9149bd2998447226bfaafb79c7d32b68b9846891e029e9152eda5aed88d"} Oct 30 13:47:27.098782 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:27.098326 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" event={"ID":"712fe9d3-fad8-4615-b9ba-18eeee7ec299","Type":"ContainerStarted","Data":"c0a3412a5ecaba1b566574315a5218a29133020f7ccbe79bfecce1826b31c552"} Oct 30 13:47:27.174953 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:27.174880 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" podStartSLOduration=2.174864037 podStartE2EDuration="2.174864037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:47:27.172145534 +0000 UTC m=+3303.804332697" watchObservedRunningTime="2026-10-30 13:47:27.174864037 +0000 UTC m=+3303.807051202" Oct 30 13:47:27.267064 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:27.267023 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:27.268820 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:27.268790 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-sm24s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 13:47:27.268966 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:27.268851 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" podUID="712fe9d3-fad8-4615-b9ba-18eeee7ec299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 13:47:28.267849 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:28.267804 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-sm24s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 13:47:28.268276 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:28.267865 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" podUID="712fe9d3-fad8-4615-b9ba-18eeee7ec299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 13:47:29.267557 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:29.267518 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-sm24s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 13:47:29.267780 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:29.267576 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" podUID="712fe9d3-fad8-4615-b9ba-18eeee7ec299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 13:47:29.423636 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:47:29.423595 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:47:30.267823 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:30.267785 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-sm24s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 13:47:30.268096 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:30.267851 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" podUID="712fe9d3-fad8-4615-b9ba-18eeee7ec299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 13:47:31.267318 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:31.267280 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-sm24s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 13:47:31.267748 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:31.267348 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" podUID="712fe9d3-fad8-4615-b9ba-18eeee7ec299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 13:47:32.267847 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:32.267805 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-sm24s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 13:47:32.268275 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:32.267874 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" podUID="712fe9d3-fad8-4615-b9ba-18eeee7ec299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 13:47:33.267960 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:33.267920 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-sm24s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 13:47:33.268382 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:33.267982 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" podUID="712fe9d3-fad8-4615-b9ba-18eeee7ec299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 13:47:34.267937 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:34.267892 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-sm24s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 13:47:34.268128 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:34.267950 7010 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" podUID="712fe9d3-fad8-4615-b9ba-18eeee7ec299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 13:47:35.288662 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:35.288627 7010 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:35.289187 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:35.288993 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:35.289808 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:35.289788 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" Oct 30 13:47:59.467919 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:47:59.467878 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:47:59.963684 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:59.963637 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-k68w2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:47:59.963874 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:47:59.963712 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" podUID="fd79a34d-b24a-41f2-b01a-55b70b9972f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:48:29.398986 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:48:29.398932 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:48:59.401914 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:48:59.401875 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:49:29.402397 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:49:29.402357 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:49:59.399720 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:49:59.399633 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:50:06.338664 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:50:06.338616 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-sm24s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:50:06.339417 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:50:06.339362 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" podUID="712fe9d3-fad8-4615-b9ba-18eeee7ec299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:50:29.400373 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:50:29.400334 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:50:59.399303 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:50:59.399259 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:51:29.401143 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:51:29.401069 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:51:59.398664 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:51:59.398618 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:52:24.079066 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:52:24.079019 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:52:24.104121 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:52:24.104095 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:52:29.421212 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:52:29.421163 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:52:59.399150 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:52:59.399101 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:53:29.399126 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:53:29.399089 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:53:59.399230 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:53:59.399184 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:54:21.335511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:54:21.334575 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-sm24s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:54:21.335511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:54:21.334655 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-sm24s" podUID="712fe9d3-fad8-4615-b9ba-18eeee7ec299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:54:29.400727 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:54:29.400684 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:54:59.399714 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:54:59.399673 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:55:29.399532 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:55:29.399476 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:55:59.405025 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:55:59.404945 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:56:29.399693 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:56:29.399650 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:56:59.398145 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:56:59.398102 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:57:24.110534 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:24.110506 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:57:24.134847 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:24.134822 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jz7rd_fce77b3b-df3e-4d04-945c-68dc11edc61f/ovn-acl-logging/0.log" Oct 30 13:57:29.398326 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:57:29.398281 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:57:39.697463 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:39.697421 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v"] Oct 30 13:57:39.698005 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:39.697740 7010 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" podUID="64d5d642-8a19-42aa-acdd-61ea5c8960bf" containerName="maas-api" containerID="cri-o://bfa922987f4a6d208b32a94fb289464020ae7721e906be1d28c96b5f37c4a1b9" gracePeriod=30 Oct 30 13:57:40.657813 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:40.657769 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" podUID="64d5d642-8a19-42aa-acdd-61ea5c8960bf" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.28:8443/health\": dial tcp 10.134.0.28:8443: connect: connection refused" Oct 30 13:57:40.942875 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:40.942849 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:57:41.064830 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.064794 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d5d642-8a19-42aa-acdd-61ea5c8960bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d5d642-8a19-42aa-acdd-61ea5c8960bf-openshift-service-ca\") pod \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\" (UID: \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\") " Oct 30 13:57:41.065018 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.064984 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d5d642-8a19-42aa-acdd-61ea5c8960bf-kube-api-access-skmwn\" (UniqueName: \"kubernetes.io/projected/64d5d642-8a19-42aa-acdd-61ea5c8960bf-kube-api-access-skmwn\") pod \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\" (UID: \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\") " Oct 30 13:57:41.065089 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.065038 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d5d642-8a19-42aa-acdd-61ea5c8960bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d5d642-8a19-42aa-acdd-61ea5c8960bf-metrics-certs\") pod \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\" (UID: \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\") " Oct 30 13:57:41.065156 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.065095 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d5d642-8a19-42aa-acdd-61ea5c8960bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d5d642-8a19-42aa-acdd-61ea5c8960bf-maas-api-tls\") pod \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\" (UID: \"64d5d642-8a19-42aa-acdd-61ea5c8960bf\") " Oct 30 13:57:41.065241 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.065213 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d5d642-8a19-42aa-acdd-61ea5c8960bf-openshift-service-ca" pod "64d5d642-8a19-42aa-acdd-61ea5c8960bf" (UID: "64d5d642-8a19-42aa-acdd-61ea5c8960bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:57:41.065420 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.065402 7010 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d5d642-8a19-42aa-acdd-61ea5c8960bf-openshift-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:57:41.067173 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.067143 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d5d642-8a19-42aa-acdd-61ea5c8960bf-kube-api-access-skmwn" pod "64d5d642-8a19-42aa-acdd-61ea5c8960bf" (UID: "64d5d642-8a19-42aa-acdd-61ea5c8960bf"). InnerVolumeSpecName "kube-api-access-skmwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:57:41.067276 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.067147 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d5d642-8a19-42aa-acdd-61ea5c8960bf-metrics-certs" pod "64d5d642-8a19-42aa-acdd-61ea5c8960bf" (UID: "64d5d642-8a19-42aa-acdd-61ea5c8960bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:57:41.067276 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.067246 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d5d642-8a19-42aa-acdd-61ea5c8960bf-maas-api-tls" pod "64d5d642-8a19-42aa-acdd-61ea5c8960bf" (UID: "64d5d642-8a19-42aa-acdd-61ea5c8960bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:57:41.166503 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.166461 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d5d642-8a19-42aa-acdd-61ea5c8960bf-maas-api-tls\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:57:41.166503 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.166507 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skmwn\" (UniqueName: \"kubernetes.io/projected/64d5d642-8a19-42aa-acdd-61ea5c8960bf-kube-api-access-skmwn\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:57:41.166690 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.166518 7010 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d5d642-8a19-42aa-acdd-61ea5c8960bf-metrics-certs\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:57:41.205350 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.205279 7010 generic.go:356] "Generic (PLEG): container finished" podID="64d5d642-8a19-42aa-acdd-61ea5c8960bf" containerID="bfa922987f4a6d208b32a94fb289464020ae7721e906be1d28c96b5f37c4a1b9" exitCode=0 Oct 30 13:57:41.205350 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.205328 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" Oct 30 13:57:41.205582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.205371 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" event={"ID":"64d5d642-8a19-42aa-acdd-61ea5c8960bf","Type":"ContainerDied","Data":"bfa922987f4a6d208b32a94fb289464020ae7721e906be1d28c96b5f37c4a1b9"} Oct 30 13:57:41.205582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.205432 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v" event={"ID":"64d5d642-8a19-42aa-acdd-61ea5c8960bf","Type":"ContainerDied","Data":"23a7e2a988a3a73db25b53095eb7ddbb65abd54573064bbd658c3124ed16edee"} Oct 30 13:57:41.205582 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.205461 7010 scope.go:122] "RemoveContainer" containerID="bfa922987f4a6d208b32a94fb289464020ae7721e906be1d28c96b5f37c4a1b9" Oct 30 13:57:41.214452 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.214436 7010 scope.go:122] "RemoveContainer" containerID="bfa922987f4a6d208b32a94fb289464020ae7721e906be1d28c96b5f37c4a1b9" Oct 30 13:57:41.214742 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:57:41.214722 7010 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa922987f4a6d208b32a94fb289464020ae7721e906be1d28c96b5f37c4a1b9\": container with ID starting with bfa922987f4a6d208b32a94fb289464020ae7721e906be1d28c96b5f37c4a1b9 not found: ID does not exist" containerID="bfa922987f4a6d208b32a94fb289464020ae7721e906be1d28c96b5f37c4a1b9" Oct 30 13:57:41.214821 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.214753 7010 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa922987f4a6d208b32a94fb289464020ae7721e906be1d28c96b5f37c4a1b9"} err="failed to get container status \"bfa922987f4a6d208b32a94fb289464020ae7721e906be1d28c96b5f37c4a1b9\": rpc error: code = NotFound desc = could not find container \"bfa922987f4a6d208b32a94fb289464020ae7721e906be1d28c96b5f37c4a1b9\": container with ID starting with bfa922987f4a6d208b32a94fb289464020ae7721e906be1d28c96b5f37c4a1b9 not found: ID does not exist" Oct 30 13:57:41.256339 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.256277 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v"] Oct 30 13:57:41.267159 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.267129 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-4zm8v"] Oct 30 13:57:41.486758 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:41.486670 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d5d642-8a19-42aa-acdd-61ea5c8960bf" path="/var/lib/kubelet/pods/64d5d642-8a19-42aa-acdd-61ea5c8960bf/volumes" Oct 30 13:57:48.398178 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.398140 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j"] Oct 30 13:57:48.398612 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.398567 7010 cpu_manager.go:414] "containerMap: removing container" podUID="64d5d642-8a19-42aa-acdd-61ea5c8960bf" containerName="maas-api" Oct 30 13:57:48.398612 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.398588 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d5d642-8a19-42aa-acdd-61ea5c8960bf" containerName="maas-api" Oct 30 13:57:48.398715 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.398691 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d5d642-8a19-42aa-acdd-61ea5c8960bf" containerName="maas-api" Oct 30 13:57:48.403130 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.403109 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j" Oct 30 13:57:48.424263 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.424239 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:57:48.435550 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.435521 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:57:48.436701 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.436676 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j"] Oct 30 13:57:48.475042 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.475017 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-sqs7j\" (UID: \"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j" Oct 30 13:57:48.475144 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.475127 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddtcr\" (UniqueName: \"kubernetes.io/projected/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-kube-api-access-ddtcr\") pod \"maas-api-revoke-sub-f03629db-sqs7j\" (UID: \"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j" Oct 30 13:57:48.475188 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.475158 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-sqs7j\" (UID: \"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j" Oct 30 13:57:48.575829 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.575800 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddtcr\" (UniqueName: \"kubernetes.io/projected/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-kube-api-access-ddtcr\") pod \"maas-api-revoke-sub-f03629db-sqs7j\" (UID: \"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j" Oct 30 13:57:48.575928 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.575838 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-sqs7j\" (UID: \"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j" Oct 30 13:57:48.575928 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.575884 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-sqs7j\" (UID: \"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j" Oct 30 13:57:48.576450 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.576433 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-sqs7j\" (UID: \"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j" Oct 30 13:57:48.578068 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.578047 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-sqs7j\" (UID: \"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j" Oct 30 13:57:48.610549 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.610521 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddtcr\" (UniqueName: \"kubernetes.io/projected/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-kube-api-access-ddtcr\") pod \"maas-api-revoke-sub-f03629db-sqs7j\" (UID: \"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j" Oct 30 13:57:48.731595 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:48.731540 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j" Oct 30 13:57:49.035673 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:49.035576 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j"] Oct 30 13:57:49.040046 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:57:49.040013 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77c9d8c_d0a0_42ed_97f9_f72dee10dac0.slice/crio-bfe268dd3033c0290ac20ff90905c67ad1d201fdc3dcf12359c0f26c74e35807 WatchSource:0}: Error finding container bfe268dd3033c0290ac20ff90905c67ad1d201fdc3dcf12359c0f26c74e35807: Status 404 returned error can't find the container with id bfe268dd3033c0290ac20ff90905c67ad1d201fdc3dcf12359c0f26c74e35807 Oct 30 13:57:49.271703 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:49.271662 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j" event={"ID":"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0","Type":"ContainerStarted","Data":"bfe268dd3033c0290ac20ff90905c67ad1d201fdc3dcf12359c0f26c74e35807"} Oct 30 13:57:50.277605 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:50.277567 7010 generic.go:356] "Generic (PLEG): container finished" podID="e77c9d8c-d0a0-42ed-97f9-f72dee10dac0" containerID="4f523fb3798921f3f62eff6232f94910c4c3e984ffe33b8d70886198862b75f1" exitCode=0 Oct 30 13:57:50.277982 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:50.277653 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j" event={"ID":"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0","Type":"ContainerDied","Data":"4f523fb3798921f3f62eff6232f94910c4c3e984ffe33b8d70886198862b75f1"} Oct 30 13:57:51.420575 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:51.420548 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j" Oct 30 13:57:51.502547 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:51.502521 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-maas-api-openshift-ca\") pod \"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0\" (UID: \"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0\") " Oct 30 13:57:51.502706 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:51.502624 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-maas-api-xks-ca\") pod \"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0\" (UID: \"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0\") " Oct 30 13:57:51.502706 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:51.502659 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-kube-api-access-ddtcr\" (UniqueName: \"kubernetes.io/projected/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-kube-api-access-ddtcr\") pod \"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0\" (UID: \"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0\") " Oct 30 13:57:51.502879 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:51.502853 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-maas-api-openshift-ca" pod "e77c9d8c-d0a0-42ed-97f9-f72dee10dac0" (UID: "e77c9d8c-d0a0-42ed-97f9-f72dee10dac0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:57:51.504743 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:51.504719 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-maas-api-xks-ca" pod "e77c9d8c-d0a0-42ed-97f9-f72dee10dac0" (UID: "e77c9d8c-d0a0-42ed-97f9-f72dee10dac0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:57:51.504843 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:51.504790 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-kube-api-access-ddtcr" pod "e77c9d8c-d0a0-42ed-97f9-f72dee10dac0" (UID: "e77c9d8c-d0a0-42ed-97f9-f72dee10dac0"). InnerVolumeSpecName "kube-api-access-ddtcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:57:51.604136 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:51.604083 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-maas-api-openshift-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:57:51.604136 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:51.604105 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-maas-api-xks-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:57:51.604136 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:51.604116 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddtcr\" (UniqueName: \"kubernetes.io/projected/e77c9d8c-d0a0-42ed-97f9-f72dee10dac0-kube-api-access-ddtcr\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:57:52.288713 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:52.288679 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j" event={"ID":"e77c9d8c-d0a0-42ed-97f9-f72dee10dac0","Type":"ContainerDied","Data":"bfe268dd3033c0290ac20ff90905c67ad1d201fdc3dcf12359c0f26c74e35807"} Oct 30 13:57:52.288886 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:52.288732 7010 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfe268dd3033c0290ac20ff90905c67ad1d201fdc3dcf12359c0f26c74e35807" Oct 30 13:57:52.288886 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:52.288691 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sqs7j" Oct 30 13:57:58.621511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.618771 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv"] Oct 30 13:57:58.621511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.619353 7010 cpu_manager.go:414] "containerMap: removing container" podUID="e77c9d8c-d0a0-42ed-97f9-f72dee10dac0" containerName="revoke-keys" Oct 30 13:57:58.621511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.619373 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77c9d8c-d0a0-42ed-97f9-f72dee10dac0" containerName="revoke-keys" Oct 30 13:57:58.621511 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.619553 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77c9d8c-d0a0-42ed-97f9-f72dee10dac0" containerName="revoke-keys" Oct 30 13:57:58.640808 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.640780 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" Oct 30 13:57:58.668512 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.647798 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv"] Oct 30 13:57:58.668982 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.662117 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:57:58.668982 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.662782 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:57:58.668982 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.665990 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12adf071-3a63-4fa1-91af-20e2459b5bba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dzgvv\" (UID: \"12adf071-3a63-4fa1-91af-20e2459b5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" Oct 30 13:57:58.668982 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.668846 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12adf071-3a63-4fa1-91af-20e2459b5bba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dzgvv\" (UID: \"12adf071-3a63-4fa1-91af-20e2459b5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" Oct 30 13:57:58.668982 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.668947 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmpzm\" (UniqueName: \"kubernetes.io/projected/12adf071-3a63-4fa1-91af-20e2459b5bba-kube-api-access-fmpzm\") pod \"maas-api-revoke-sub-0165d7d2-dzgvv\" (UID: \"12adf071-3a63-4fa1-91af-20e2459b5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" Oct 30 13:57:58.770769 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.769863 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmpzm\" (UniqueName: \"kubernetes.io/projected/12adf071-3a63-4fa1-91af-20e2459b5bba-kube-api-access-fmpzm\") pod \"maas-api-revoke-sub-0165d7d2-dzgvv\" (UID: \"12adf071-3a63-4fa1-91af-20e2459b5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" Oct 30 13:57:58.770769 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.769922 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12adf071-3a63-4fa1-91af-20e2459b5bba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dzgvv\" (UID: \"12adf071-3a63-4fa1-91af-20e2459b5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" Oct 30 13:57:58.770769 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.769962 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12adf071-3a63-4fa1-91af-20e2459b5bba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dzgvv\" (UID: \"12adf071-3a63-4fa1-91af-20e2459b5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" Oct 30 13:57:58.771193 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.771161 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12adf071-3a63-4fa1-91af-20e2459b5bba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dzgvv\" (UID: \"12adf071-3a63-4fa1-91af-20e2459b5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" Oct 30 13:57:58.773211 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.773175 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12adf071-3a63-4fa1-91af-20e2459b5bba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dzgvv\" (UID: \"12adf071-3a63-4fa1-91af-20e2459b5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" Oct 30 13:57:58.859818 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.859534 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmpzm\" (UniqueName: \"kubernetes.io/projected/12adf071-3a63-4fa1-91af-20e2459b5bba-kube-api-access-fmpzm\") pod \"maas-api-revoke-sub-0165d7d2-dzgvv\" (UID: \"12adf071-3a63-4fa1-91af-20e2459b5bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" Oct 30 13:57:58.975259 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:58.975172 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" Oct 30 13:57:59.468404 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:59.468096 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv"] Oct 30 13:57:59.471411 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:57:59.471352 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:57:59.472453 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:57:59.472420 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12adf071_3a63_4fa1_91af_20e2459b5bba.slice/crio-d48e2b796873502431c6d748e34c5624eecadfb65c81c559f109afc56c6b3ae4 WatchSource:0}: Error finding container d48e2b796873502431c6d748e34c5624eecadfb65c81c559f109afc56c6b3ae4: Status 404 returned error can't find the container with id d48e2b796873502431c6d748e34c5624eecadfb65c81c559f109afc56c6b3ae4 Oct 30 13:57:59.967683 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:59.967643 7010 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-64d4cc998c-k68w2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 13:57:59.968030 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:57:59.967690 7010 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-64d4cc998c-k68w2" podUID="fd79a34d-b24a-41f2-b01a-55b70b9972f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:58:00.346176 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:00.346134 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" event={"ID":"12adf071-3a63-4fa1-91af-20e2459b5bba","Type":"ContainerStarted","Data":"960a79a438a07a130dbde13461ffc4e268400690e7f288fee2651c7f01e9dd4a"} Oct 30 13:58:00.346358 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:00.346193 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" event={"ID":"12adf071-3a63-4fa1-91af-20e2459b5bba","Type":"ContainerStarted","Data":"d48e2b796873502431c6d748e34c5624eecadfb65c81c559f109afc56c6b3ae4"} Oct 30 13:58:00.385986 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:00.385933 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" podStartSLOduration=2.385919999 podStartE2EDuration="2.385919999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:00.384258771 +0000 UTC m=+3937.016445942" watchObservedRunningTime="2026-10-30 13:58:00.385919999 +0000 UTC m=+3937.018107160" Oct 30 13:58:01.352520 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:01.352470 7010 generic.go:356] "Generic (PLEG): container finished" podID="12adf071-3a63-4fa1-91af-20e2459b5bba" containerID="960a79a438a07a130dbde13461ffc4e268400690e7f288fee2651c7f01e9dd4a" exitCode=0 Oct 30 13:58:01.352913 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:01.352546 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" event={"ID":"12adf071-3a63-4fa1-91af-20e2459b5bba","Type":"ContainerDied","Data":"960a79a438a07a130dbde13461ffc4e268400690e7f288fee2651c7f01e9dd4a"} Oct 30 13:58:02.493513 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:02.493476 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" Oct 30 13:58:02.515362 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:02.515337 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12adf071-3a63-4fa1-91af-20e2459b5bba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12adf071-3a63-4fa1-91af-20e2459b5bba-maas-api-xks-ca\") pod \"12adf071-3a63-4fa1-91af-20e2459b5bba\" (UID: \"12adf071-3a63-4fa1-91af-20e2459b5bba\") " Oct 30 13:58:02.515461 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:02.515399 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12adf071-3a63-4fa1-91af-20e2459b5bba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12adf071-3a63-4fa1-91af-20e2459b5bba-maas-api-openshift-ca\") pod \"12adf071-3a63-4fa1-91af-20e2459b5bba\" (UID: \"12adf071-3a63-4fa1-91af-20e2459b5bba\") " Oct 30 13:58:02.515557 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:02.515477 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12adf071-3a63-4fa1-91af-20e2459b5bba-kube-api-access-fmpzm\" (UniqueName: \"kubernetes.io/projected/12adf071-3a63-4fa1-91af-20e2459b5bba-kube-api-access-fmpzm\") pod \"12adf071-3a63-4fa1-91af-20e2459b5bba\" (UID: \"12adf071-3a63-4fa1-91af-20e2459b5bba\") " Oct 30 13:58:02.515787 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:02.515765 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12adf071-3a63-4fa1-91af-20e2459b5bba-maas-api-openshift-ca" pod "12adf071-3a63-4fa1-91af-20e2459b5bba" (UID: "12adf071-3a63-4fa1-91af-20e2459b5bba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:58:02.517275 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:02.517253 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12adf071-3a63-4fa1-91af-20e2459b5bba-maas-api-xks-ca" pod "12adf071-3a63-4fa1-91af-20e2459b5bba" (UID: "12adf071-3a63-4fa1-91af-20e2459b5bba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:58:02.517417 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:02.517393 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12adf071-3a63-4fa1-91af-20e2459b5bba-kube-api-access-fmpzm" pod "12adf071-3a63-4fa1-91af-20e2459b5bba" (UID: "12adf071-3a63-4fa1-91af-20e2459b5bba"). InnerVolumeSpecName "kube-api-access-fmpzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:58:02.616650 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:02.616589 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmpzm\" (UniqueName: \"kubernetes.io/projected/12adf071-3a63-4fa1-91af-20e2459b5bba-kube-api-access-fmpzm\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:58:02.616650 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:02.616611 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12adf071-3a63-4fa1-91af-20e2459b5bba-maas-api-xks-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:58:02.616650 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:02.616622 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12adf071-3a63-4fa1-91af-20e2459b5bba-maas-api-openshift-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:58:03.362986 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:03.362956 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" Oct 30 13:58:03.363164 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:03.362949 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dzgvv" event={"ID":"12adf071-3a63-4fa1-91af-20e2459b5bba","Type":"ContainerDied","Data":"d48e2b796873502431c6d748e34c5624eecadfb65c81c559f109afc56c6b3ae4"} Oct 30 13:58:03.363164 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:03.363083 7010 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d48e2b796873502431c6d748e34c5624eecadfb65c81c559f109afc56c6b3ae4" Oct 30 13:58:09.576742 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.576696 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4"] Oct 30 13:58:09.577246 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.577165 7010 cpu_manager.go:414] "containerMap: removing container" podUID="12adf071-3a63-4fa1-91af-20e2459b5bba" containerName="revoke-keys" Oct 30 13:58:09.577246 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.577185 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12adf071-3a63-4fa1-91af-20e2459b5bba" containerName="revoke-keys" Oct 30 13:58:09.577364 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.577271 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="12adf071-3a63-4fa1-91af-20e2459b5bba" containerName="revoke-keys" Oct 30 13:58:09.580216 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.580197 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4" Oct 30 13:58:09.602563 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.602537 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4"] Oct 30 13:58:09.633101 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.633067 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 13:58:09.633288 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.633125 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 13:58:09.676581 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.676550 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cdjr4\" (UID: \"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4" Oct 30 13:58:09.676739 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.676601 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xds2n\" (UniqueName: \"kubernetes.io/projected/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-kube-api-access-xds2n\") pod \"maas-api-revoke-sub-6b19338f-cdjr4\" (UID: \"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4" Oct 30 13:58:09.676817 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.676730 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cdjr4\" (UID: \"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4" Oct 30 13:58:09.777538 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.777507 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cdjr4\" (UID: \"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4" Oct 30 13:58:09.777670 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.777559 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xds2n\" (UniqueName: \"kubernetes.io/projected/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-kube-api-access-xds2n\") pod \"maas-api-revoke-sub-6b19338f-cdjr4\" (UID: \"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4" Oct 30 13:58:09.777670 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.777585 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cdjr4\" (UID: \"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4" Oct 30 13:58:09.778220 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.778196 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cdjr4\" (UID: \"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4" Oct 30 13:58:09.779555 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.779534 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cdjr4\" (UID: \"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4" Oct 30 13:58:09.801286 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.801258 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xds2n\" (UniqueName: \"kubernetes.io/projected/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-kube-api-access-xds2n\") pod \"maas-api-revoke-sub-6b19338f-cdjr4\" (UID: \"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4" Oct 30 13:58:09.894347 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:09.894317 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4" Oct 30 13:58:10.195515 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:10.195420 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4"] Oct 30 13:58:10.200425 ip-10-0-138-103 kubenswrapper[7010]: W1030 13:58:10.200396 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e0aef4_9c9e_4ba3_b18b_ef51e4288eab.slice/crio-440770302ee2b746207d08c1a267bbf6825c5362b0cc31b547ed1a93cd47af5c WatchSource:0}: Error finding container 440770302ee2b746207d08c1a267bbf6825c5362b0cc31b547ed1a93cd47af5c: Status 404 returned error can't find the container with id 440770302ee2b746207d08c1a267bbf6825c5362b0cc31b547ed1a93cd47af5c Oct 30 13:58:10.396686 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:10.396658 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4" event={"ID":"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab","Type":"ContainerStarted","Data":"440770302ee2b746207d08c1a267bbf6825c5362b0cc31b547ed1a93cd47af5c"} Oct 30 13:58:11.402945 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:11.402914 7010 generic.go:356] "Generic (PLEG): container finished" podID="a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab" containerID="51ff1a43ae762385d75bb8a32c19e1ec6434804a02fa6aa5e719ed3b63fba876" exitCode=0 Oct 30 13:58:11.403368 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:11.402990 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4" event={"ID":"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab","Type":"ContainerDied","Data":"51ff1a43ae762385d75bb8a32c19e1ec6434804a02fa6aa5e719ed3b63fba876"} Oct 30 13:58:12.531296 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:12.531271 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4" Oct 30 13:58:12.600717 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:12.600683 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-maas-api-openshift-ca\") pod \"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab\" (UID: \"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab\") " Oct 30 13:58:12.600876 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:12.600786 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-kube-api-access-xds2n\" (UniqueName: \"kubernetes.io/projected/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-kube-api-access-xds2n\") pod \"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab\" (UID: \"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab\") " Oct 30 13:58:12.600876 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:12.600855 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-maas-api-xks-ca\") pod \"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab\" (UID: \"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab\") " Oct 30 13:58:12.601093 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:12.601068 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-maas-api-openshift-ca" pod "a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab" (UID: "a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:58:12.601181 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:12.601165 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-maas-api-openshift-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:58:12.602875 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:12.602846 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-kube-api-access-xds2n" pod "a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab" (UID: "a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab"). InnerVolumeSpecName "kube-api-access-xds2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:58:12.602975 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:12.602888 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-maas-api-xks-ca" pod "a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab" (UID: "a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:58:12.702102 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:12.702049 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-maas-api-xks-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:58:12.702102 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:12.702070 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xds2n\" (UniqueName: \"kubernetes.io/projected/a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab-kube-api-access-xds2n\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 13:58:13.412593 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:13.412561 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4" Oct 30 13:58:13.412593 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:13.412587 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cdjr4" event={"ID":"a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab","Type":"ContainerDied","Data":"440770302ee2b746207d08c1a267bbf6825c5362b0cc31b547ed1a93cd47af5c"} Oct 30 13:58:13.412866 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:13.412632 7010 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="440770302ee2b746207d08c1a267bbf6825c5362b0cc31b547ed1a93cd47af5c" Oct 30 13:58:29.399264 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:58:29.399155 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:58:32.421942 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:32.421912 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-59xk4_6c0c7b30-b37a-471f-bab0-fcbfcdefa069/manager/0.log" Oct 30 13:58:33.705160 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:33.705135 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-678d69fff5-m9kz4_40b0026f-ae61-4211-b0f3-18e32fe5b20f/copy-manifests/0.log" Oct 30 13:58:33.737091 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:33.737061 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-678d69fff5-m9kz4_40b0026f-ae61-4211-b0f3-18e32fe5b20f/manager/0.log" Oct 30 13:58:35.641106 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:35.641076 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-dzgvv_12adf071-3a63-4fa1-91af-20e2459b5bba/revoke-keys/0.log" Oct 30 13:58:36.392865 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:36.392835 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-cdjr4_a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab/revoke-keys/0.log" Oct 30 13:58:36.942729 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:36.942704 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-sqs7j_e77c9d8c-d0a0-42ed-97f9-f72dee10dac0/revoke-keys/0.log" Oct 30 13:58:37.321686 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:37.321612 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-qr9vn_4a989e02-7066-4728-ad39-843d52c81703/postgres/0.log" Oct 30 13:58:39.506666 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:39.506641 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rzvww_7931e956-afe1-40cb-93b2-d06d02c0b35a/kuadrant-console-plugin/0.log" Oct 30 13:58:41.129748 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:41.129714 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8nb8t_e3b65b07-570d-49f3-8b74-c613b6e8c669/discovery/0.log" Oct 30 13:58:41.549553 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:41.549457 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65dc4d65b9-cvtrj_ea775e41-0676-42f8-937c-5b771342e724/kube-auth-proxy/0.log" Oct 30 13:58:42.560930 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:42.560897 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-k68w2_fd79a34d-b24a-41f2-b01a-55b70b9972f2/istio-proxy/0.log" Oct 30 13:58:42.771386 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:42.771356 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-sm24s_712fe9d3-fad8-4615-b9ba-18eeee7ec299/istio-proxy/0.log" Oct 30 13:58:43.185586 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:43.185554 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-tlqh6_3c7855fd-e6a6-44ad-870e-ed9f483528f2/istio-proxy/0.log" Oct 30 13:58:57.080351 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:57.080321 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-59xk4_6c0c7b30-b37a-471f-bab0-fcbfcdefa069/manager/0.log" Oct 30 13:58:58.232192 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:58.232161 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-678d69fff5-m9kz4_40b0026f-ae61-4211-b0f3-18e32fe5b20f/copy-manifests/0.log" Oct 30 13:58:58.262133 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:58:58.262101 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-678d69fff5-m9kz4_40b0026f-ae61-4211-b0f3-18e32fe5b20f/manager/0.log" Oct 30 13:58:59.398400 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:58:59.398362 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:59:00.013298 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:59:00.013254 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-dzgvv_12adf071-3a63-4fa1-91af-20e2459b5bba/revoke-keys/0.log" Oct 30 13:59:00.755555 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:59:00.755524 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-cdjr4_a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab/revoke-keys/0.log" Oct 30 13:59:01.360595 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:59:01.360567 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-sqs7j_e77c9d8c-d0a0-42ed-97f9-f72dee10dac0/revoke-keys/0.log" Oct 30 13:59:01.839013 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:59:01.838980 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-qr9vn_4a989e02-7066-4728-ad39-843d52c81703/postgres/0.log" Oct 30 13:59:03.689468 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:59:03.689442 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rzvww_7931e956-afe1-40cb-93b2-d06d02c0b35a/kuadrant-console-plugin/0.log" Oct 30 13:59:05.111587 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:59:05.111551 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8nb8t_e3b65b07-570d-49f3-8b74-c613b6e8c669/discovery/0.log" Oct 30 13:59:05.540614 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:59:05.540577 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65dc4d65b9-cvtrj_ea775e41-0676-42f8-937c-5b771342e724/kube-auth-proxy/0.log" Oct 30 13:59:06.576144 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:59:06.576098 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-k68w2_fd79a34d-b24a-41f2-b01a-55b70b9972f2/istio-proxy/0.log" Oct 30 13:59:06.788962 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:59:06.788929 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-sm24s_712fe9d3-fad8-4615-b9ba-18eeee7ec299/istio-proxy/0.log" Oct 30 13:59:07.196967 ip-10-0-138-103 kubenswrapper[7010]: I1030 13:59:07.196934 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-tlqh6_3c7855fd-e6a6-44ad-870e-ed9f483528f2/istio-proxy/0.log" Oct 30 13:59:29.399288 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:59:29.399235 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 13:59:59.401325 ip-10-0-138-103 kubenswrapper[7010]: E1030 13:59:59.401278 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 14:00:00.158952 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:00.158913 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh"] Oct 30 14:00:00.159340 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:00.159321 7010 cpu_manager.go:414] "containerMap: removing container" podUID="a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab" containerName="revoke-keys" Oct 30 14:00:00.159340 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:00.159336 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab" containerName="revoke-keys" Oct 30 14:00:00.159522 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:00.159394 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e0aef4-9c9e-4ba3-b18b-ef51e4288eab" containerName="revoke-keys" Oct 30 14:00:00.162529 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:00.162510 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" Oct 30 14:00:00.176735 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:00.176712 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t46mn\"" Oct 30 14:00:00.251043 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:00.251008 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfk7n\" (UniqueName: \"kubernetes.io/projected/42907fc3-d91d-4c89-9689-15dd4cc98e24-kube-api-access-lfk7n\") pod \"maas-api-key-cleanup-29889480-dwqfh\" (UID: \"42907fc3-d91d-4c89-9689-15dd4cc98e24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" Oct 30 14:00:00.251043 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:00.251047 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42907fc3-d91d-4c89-9689-15dd4cc98e24-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889480-dwqfh\" (UID: \"42907fc3-d91d-4c89-9689-15dd4cc98e24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" Oct 30 14:00:00.258384 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:00.258356 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh"] Oct 30 14:00:00.351944 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:00.351903 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfk7n\" (UniqueName: \"kubernetes.io/projected/42907fc3-d91d-4c89-9689-15dd4cc98e24-kube-api-access-lfk7n\") pod \"maas-api-key-cleanup-29889480-dwqfh\" (UID: \"42907fc3-d91d-4c89-9689-15dd4cc98e24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" Oct 30 14:00:00.351944 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:00.351948 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42907fc3-d91d-4c89-9689-15dd4cc98e24-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889480-dwqfh\" (UID: \"42907fc3-d91d-4c89-9689-15dd4cc98e24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" Oct 30 14:00:00.352597 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:00.352577 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42907fc3-d91d-4c89-9689-15dd4cc98e24-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889480-dwqfh\" (UID: \"42907fc3-d91d-4c89-9689-15dd4cc98e24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" Oct 30 14:00:00.374562 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:00.374527 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfk7n\" (UniqueName: \"kubernetes.io/projected/42907fc3-d91d-4c89-9689-15dd4cc98e24-kube-api-access-lfk7n\") pod \"maas-api-key-cleanup-29889480-dwqfh\" (UID: \"42907fc3-d91d-4c89-9689-15dd4cc98e24\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" Oct 30 14:00:00.474752 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:00.474666 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" Oct 30 14:00:00.772424 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:00.772336 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh"] Oct 30 14:00:00.777255 ip-10-0-138-103 kubenswrapper[7010]: W1030 14:00:00.777221 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42907fc3_d91d_4c89_9689_15dd4cc98e24.slice/crio-2e61282fcdaa0af12282b43853a99b9f6727a0c6f456fd0e4450a1b6129535fe WatchSource:0}: Error finding container 2e61282fcdaa0af12282b43853a99b9f6727a0c6f456fd0e4450a1b6129535fe: Status 404 returned error can't find the container with id 2e61282fcdaa0af12282b43853a99b9f6727a0c6f456fd0e4450a1b6129535fe Oct 30 14:00:00.912936 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:00.912896 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" event={"ID":"42907fc3-d91d-4c89-9689-15dd4cc98e24","Type":"ContainerStarted","Data":"2e61282fcdaa0af12282b43853a99b9f6727a0c6f456fd0e4450a1b6129535fe"} Oct 30 14:00:01.918984 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:01.918952 7010 generic.go:356] "Generic (PLEG): container finished" podID="42907fc3-d91d-4c89-9689-15dd4cc98e24" containerID="d94ee0296ea2c149c430be1df66f6e0e892e331c73c1641bea8fb792de303b49" exitCode=60 Oct 30 14:00:01.919399 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:01.919004 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" event={"ID":"42907fc3-d91d-4c89-9689-15dd4cc98e24","Type":"ContainerDied","Data":"d94ee0296ea2c149c430be1df66f6e0e892e331c73c1641bea8fb792de303b49"} Oct 30 14:00:01.919399 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:01.919265 7010 scope.go:122] "RemoveContainer" containerID="d94ee0296ea2c149c430be1df66f6e0e892e331c73c1641bea8fb792de303b49" Oct 30 14:00:02.925457 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:02.925421 7010 generic.go:356] "Generic (PLEG): container finished" podID="42907fc3-d91d-4c89-9689-15dd4cc98e24" containerID="113250ba10dacdf30046a5fcf175cde2f1cffdea1183aef27dd2b7674d22e345" exitCode=60 Oct 30 14:00:02.925874 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:02.925516 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" event={"ID":"42907fc3-d91d-4c89-9689-15dd4cc98e24","Type":"ContainerDied","Data":"113250ba10dacdf30046a5fcf175cde2f1cffdea1183aef27dd2b7674d22e345"} Oct 30 14:00:02.925874 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:02.925576 7010 scope.go:122] "RemoveContainer" containerID="d94ee0296ea2c149c430be1df66f6e0e892e331c73c1641bea8fb792de303b49" Oct 30 14:00:02.925969 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:02.925884 7010 scope.go:122] "RemoveContainer" containerID="113250ba10dacdf30046a5fcf175cde2f1cffdea1183aef27dd2b7674d22e345" Oct 30 14:00:02.926136 ip-10-0-138-103 kubenswrapper[7010]: E1030 14:00:02.926117 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889480-dwqfh_odh-ai-gateway-infra(42907fc3-d91d-4c89-9689-15dd4cc98e24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" podUID="42907fc3-d91d-4c89-9689-15dd4cc98e24" Oct 30 14:00:03.931356 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:03.931322 7010 scope.go:122] "RemoveContainer" containerID="113250ba10dacdf30046a5fcf175cde2f1cffdea1183aef27dd2b7674d22e345" Oct 30 14:00:03.931869 ip-10-0-138-103 kubenswrapper[7010]: E1030 14:00:03.931676 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889480-dwqfh_odh-ai-gateway-infra(42907fc3-d91d-4c89-9689-15dd4cc98e24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" podUID="42907fc3-d91d-4c89-9689-15dd4cc98e24" Oct 30 14:00:15.481427 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:15.481384 7010 scope.go:122] "RemoveContainer" containerID="113250ba10dacdf30046a5fcf175cde2f1cffdea1183aef27dd2b7674d22e345" Oct 30 14:00:15.988276 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:15.988237 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" event={"ID":"42907fc3-d91d-4c89-9689-15dd4cc98e24","Type":"ContainerStarted","Data":"0f15727eca474458be65a1873917573595854272648b6faf8b7680d6a34fc6c3"} Oct 30 14:00:16.028809 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:16.028749 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" podStartSLOduration=16.028735576 podStartE2EDuration="16.028735576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:16.027970457 +0000 UTC m=+4072.660157617" watchObservedRunningTime="2026-10-30 14:00:16.028735576 +0000 UTC m=+4072.660922731" Oct 30 14:00:16.995200 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:16.995162 7010 generic.go:356] "Generic (PLEG): container finished" podID="42907fc3-d91d-4c89-9689-15dd4cc98e24" containerID="0f15727eca474458be65a1873917573595854272648b6faf8b7680d6a34fc6c3" exitCode=60 Oct 30 14:00:16.995708 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:16.995232 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" event={"ID":"42907fc3-d91d-4c89-9689-15dd4cc98e24","Type":"ContainerDied","Data":"0f15727eca474458be65a1873917573595854272648b6faf8b7680d6a34fc6c3"} Oct 30 14:00:16.995708 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:16.995292 7010 scope.go:122] "RemoveContainer" containerID="113250ba10dacdf30046a5fcf175cde2f1cffdea1183aef27dd2b7674d22e345" Oct 30 14:00:16.995708 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:16.995647 7010 scope.go:122] "RemoveContainer" containerID="0f15727eca474458be65a1873917573595854272648b6faf8b7680d6a34fc6c3" Oct 30 14:00:16.995940 ip-10-0-138-103 kubenswrapper[7010]: E1030 14:00:16.995921 7010 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889480-dwqfh_odh-ai-gateway-infra(42907fc3-d91d-4c89-9689-15dd4cc98e24)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" podUID="42907fc3-d91d-4c89-9689-15dd4cc98e24" Oct 30 14:00:17.050263 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:17.050227 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh"] Oct 30 14:00:18.144006 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:18.143976 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" Oct 30 14:00:18.222807 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:18.222772 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42907fc3-d91d-4c89-9689-15dd4cc98e24-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42907fc3-d91d-4c89-9689-15dd4cc98e24-maas-api-service-ca\") pod \"42907fc3-d91d-4c89-9689-15dd4cc98e24\" (UID: \"42907fc3-d91d-4c89-9689-15dd4cc98e24\") " Oct 30 14:00:18.223010 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:18.222835 7010 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42907fc3-d91d-4c89-9689-15dd4cc98e24-kube-api-access-lfk7n\" (UniqueName: \"kubernetes.io/projected/42907fc3-d91d-4c89-9689-15dd4cc98e24-kube-api-access-lfk7n\") pod \"42907fc3-d91d-4c89-9689-15dd4cc98e24\" (UID: \"42907fc3-d91d-4c89-9689-15dd4cc98e24\") " Oct 30 14:00:18.223163 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:18.223139 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42907fc3-d91d-4c89-9689-15dd4cc98e24-maas-api-service-ca" pod "42907fc3-d91d-4c89-9689-15dd4cc98e24" (UID: "42907fc3-d91d-4c89-9689-15dd4cc98e24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:00:18.225040 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:18.225008 7010 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42907fc3-d91d-4c89-9689-15dd4cc98e24-kube-api-access-lfk7n" pod "42907fc3-d91d-4c89-9689-15dd4cc98e24" (UID: "42907fc3-d91d-4c89-9689-15dd4cc98e24"). InnerVolumeSpecName "kube-api-access-lfk7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:18.324237 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:18.324139 7010 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42907fc3-d91d-4c89-9689-15dd4cc98e24-maas-api-service-ca\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 14:00:18.324237 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:18.324179 7010 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfk7n\" (UniqueName: \"kubernetes.io/projected/42907fc3-d91d-4c89-9689-15dd4cc98e24-kube-api-access-lfk7n\") on node \"ip-10-0-138-103.ec2.internal\" DevicePath \"\"" Oct 30 14:00:19.007604 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:19.007564 7010 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" Oct 30 14:00:19.007832 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:19.007608 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh" event={"ID":"42907fc3-d91d-4c89-9689-15dd4cc98e24","Type":"ContainerDied","Data":"2e61282fcdaa0af12282b43853a99b9f6727a0c6f456fd0e4450a1b6129535fe"} Oct 30 14:00:19.007832 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:19.007671 7010 scope.go:122] "RemoveContainer" containerID="0f15727eca474458be65a1873917573595854272648b6faf8b7680d6a34fc6c3" Oct 30 14:00:19.056422 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:19.056382 7010 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh"] Oct 30 14:00:19.067300 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:19.067266 7010 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889480-dwqfh"] Oct 30 14:00:19.487173 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:19.487135 7010 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42907fc3-d91d-4c89-9689-15dd4cc98e24" path="/var/lib/kubelet/pods/42907fc3-d91d-4c89-9689-15dd4cc98e24/volumes" Oct 30 14:00:29.400016 ip-10-0-138-103 kubenswrapper[7010]: E1030 14:00:29.399976 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 14:00:38.441037 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:38.441006 7010 ???:1] "http: TLS handshake error from 10.0.133.61:60228: EOF" Oct 30 14:00:38.442461 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:38.442434 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2xg75_0b497ad2-5bd4-4b7b-881d-d94c15fda49c/global-pull-secret-syncer/0.log" Oct 30 14:00:39.885679 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:39.885642 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zchsd_741107a5-c4db-4ba9-929c-58952377929f/konnectivity-agent/0.log" Oct 30 14:00:40.282247 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:40.282168 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-103.ec2.internal_4db7be0b2dbb4b62fd854fb3bbef0e8f/haproxy/0.log" Oct 30 14:00:46.700394 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:46.700367 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rzvww_7931e956-afe1-40cb-93b2-d06d02c0b35a/kuadrant-console-plugin/0.log" Oct 30 14:00:47.790536 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.790482 7010 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr"] Oct 30 14:00:47.790986 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.790896 7010 cpu_manager.go:414] "containerMap: removing container" podUID="42907fc3-d91d-4c89-9689-15dd4cc98e24" containerName="cleanup" Oct 30 14:00:47.790986 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.790911 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42907fc3-d91d-4c89-9689-15dd4cc98e24" containerName="cleanup" Oct 30 14:00:47.790986 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.790957 7010 cpu_manager.go:414] "containerMap: removing container" podUID="42907fc3-d91d-4c89-9689-15dd4cc98e24" containerName="cleanup" Oct 30 14:00:47.790986 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.790962 7010 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42907fc3-d91d-4c89-9689-15dd4cc98e24" containerName="cleanup" Oct 30 14:00:47.791229 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.791026 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="42907fc3-d91d-4c89-9689-15dd4cc98e24" containerName="cleanup" Oct 30 14:00:47.791229 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.791035 7010 memory_manager.go:365] "RemoveStaleState removing state" podUID="42907fc3-d91d-4c89-9689-15dd4cc98e24" containerName="cleanup" Oct 30 14:00:47.794282 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.794263 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:47.814621 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.814586 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr"] Oct 30 14:00:47.894327 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.894291 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rkb8w\"/\"kube-root-ca.crt\"" Oct 30 14:00:47.894535 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.894350 7010 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rkb8w\"/\"openshift-service-ca.crt\"" Oct 30 14:00:47.896200 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.896186 7010 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rkb8w\"/\"default-dockercfg-dpzm8\"" Oct 30 14:00:47.896351 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.896329 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0b7d3700-81c9-4180-b6d1-ab38139957f3-podres\") pod \"perf-node-gather-daemonset-khfdr\" (UID: \"0b7d3700-81c9-4180-b6d1-ab38139957f3\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:47.896456 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.896439 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b7d3700-81c9-4180-b6d1-ab38139957f3-sys\") pod \"perf-node-gather-daemonset-khfdr\" (UID: \"0b7d3700-81c9-4180-b6d1-ab38139957f3\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:47.896548 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.896534 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b7d3700-81c9-4180-b6d1-ab38139957f3-lib-modules\") pod \"perf-node-gather-daemonset-khfdr\" (UID: \"0b7d3700-81c9-4180-b6d1-ab38139957f3\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:47.896590 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.896555 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c55j9\" (UniqueName: \"kubernetes.io/projected/0b7d3700-81c9-4180-b6d1-ab38139957f3-kube-api-access-c55j9\") pod \"perf-node-gather-daemonset-khfdr\" (UID: \"0b7d3700-81c9-4180-b6d1-ab38139957f3\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:47.896590 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.896581 7010 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0b7d3700-81c9-4180-b6d1-ab38139957f3-proc\") pod \"perf-node-gather-daemonset-khfdr\" (UID: \"0b7d3700-81c9-4180-b6d1-ab38139957f3\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:47.997121 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.997076 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b7d3700-81c9-4180-b6d1-ab38139957f3-lib-modules\") pod \"perf-node-gather-daemonset-khfdr\" (UID: \"0b7d3700-81c9-4180-b6d1-ab38139957f3\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:47.997121 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.997114 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c55j9\" (UniqueName: \"kubernetes.io/projected/0b7d3700-81c9-4180-b6d1-ab38139957f3-kube-api-access-c55j9\") pod \"perf-node-gather-daemonset-khfdr\" (UID: \"0b7d3700-81c9-4180-b6d1-ab38139957f3\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:47.997417 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.997140 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0b7d3700-81c9-4180-b6d1-ab38139957f3-proc\") pod \"perf-node-gather-daemonset-khfdr\" (UID: \"0b7d3700-81c9-4180-b6d1-ab38139957f3\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:47.997417 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.997173 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0b7d3700-81c9-4180-b6d1-ab38139957f3-podres\") pod \"perf-node-gather-daemonset-khfdr\" (UID: \"0b7d3700-81c9-4180-b6d1-ab38139957f3\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:47.997417 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.997220 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b7d3700-81c9-4180-b6d1-ab38139957f3-lib-modules\") pod \"perf-node-gather-daemonset-khfdr\" (UID: \"0b7d3700-81c9-4180-b6d1-ab38139957f3\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:47.997417 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.997242 7010 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b7d3700-81c9-4180-b6d1-ab38139957f3-sys\") pod \"perf-node-gather-daemonset-khfdr\" (UID: \"0b7d3700-81c9-4180-b6d1-ab38139957f3\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:47.997417 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.997297 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0b7d3700-81c9-4180-b6d1-ab38139957f3-proc\") pod \"perf-node-gather-daemonset-khfdr\" (UID: \"0b7d3700-81c9-4180-b6d1-ab38139957f3\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:47.997417 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.997315 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0b7d3700-81c9-4180-b6d1-ab38139957f3-podres\") pod \"perf-node-gather-daemonset-khfdr\" (UID: \"0b7d3700-81c9-4180-b6d1-ab38139957f3\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:47.997417 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:47.997344 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b7d3700-81c9-4180-b6d1-ab38139957f3-sys\") pod \"perf-node-gather-daemonset-khfdr\" (UID: \"0b7d3700-81c9-4180-b6d1-ab38139957f3\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:48.021157 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:48.021067 7010 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c55j9\" (UniqueName: \"kubernetes.io/projected/0b7d3700-81c9-4180-b6d1-ab38139957f3-kube-api-access-c55j9\") pod \"perf-node-gather-daemonset-khfdr\" (UID: \"0b7d3700-81c9-4180-b6d1-ab38139957f3\") " pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:48.107300 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:48.107199 7010 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:48.407983 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:48.407940 7010 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr"] Oct 30 14:00:48.412357 ip-10-0-138-103 kubenswrapper[7010]: W1030 14:00:48.412325 7010 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0b7d3700_81c9_4180_b6d1_ab38139957f3.slice/crio-8706c9ddcf9f9013c7ce6960daa76c15b9217395fdf1825ebdbf2a508c5de3d3 WatchSource:0}: Error finding container 8706c9ddcf9f9013c7ce6960daa76c15b9217395fdf1825ebdbf2a508c5de3d3: Status 404 returned error can't find the container with id 8706c9ddcf9f9013c7ce6960daa76c15b9217395fdf1825ebdbf2a508c5de3d3 Oct 30 14:00:48.740729 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:48.740644 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_143b92f5-8e39-431f-aaef-edcbc459acf5/alertmanager/0.log" Oct 30 14:00:48.870564 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:48.870534 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_143b92f5-8e39-431f-aaef-edcbc459acf5/config-reloader/0.log" Oct 30 14:00:48.999666 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:48.999580 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_143b92f5-8e39-431f-aaef-edcbc459acf5/kube-rbac-proxy-web/0.log" Oct 30 14:00:49.127540 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:49.127513 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_143b92f5-8e39-431f-aaef-edcbc459acf5/kube-rbac-proxy/0.log" Oct 30 14:00:49.146717 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:49.146682 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" event={"ID":"0b7d3700-81c9-4180-b6d1-ab38139957f3","Type":"ContainerStarted","Data":"667a93c9916a07ad60e2bf15b362ef0fed1666510eebfa889394e34b64808e42"} Oct 30 14:00:49.146902 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:49.146748 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:49.146902 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:49.146770 7010 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" event={"ID":"0b7d3700-81c9-4180-b6d1-ab38139957f3","Type":"ContainerStarted","Data":"8706c9ddcf9f9013c7ce6960daa76c15b9217395fdf1825ebdbf2a508c5de3d3"} Oct 30 14:00:49.184838 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:49.184764 7010 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" podStartSLOduration=2.18474972 podStartE2EDuration="2.18474972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:49.184445557 +0000 UTC m=+4105.816632718" watchObservedRunningTime="2026-10-30 14:00:49.18474972 +0000 UTC m=+4105.816936882" Oct 30 14:00:49.256577 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:49.256501 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_143b92f5-8e39-431f-aaef-edcbc459acf5/kube-rbac-proxy-metric/0.log" Oct 30 14:00:49.383792 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:49.383766 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_143b92f5-8e39-431f-aaef-edcbc459acf5/prom-label-proxy/0.log" Oct 30 14:00:49.514242 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:49.514147 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_143b92f5-8e39-431f-aaef-edcbc459acf5/init-config-reloader/0.log" Oct 30 14:00:49.788611 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:49.788523 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-q2bdf_b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1/kube-state-metrics/0.log" Oct 30 14:00:49.908284 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:49.908250 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-q2bdf_b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1/kube-rbac-proxy-main/0.log" Oct 30 14:00:50.036524 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:50.036471 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-q2bdf_b6a0e1e1-6da5-4e81-8e1f-ac1643fd0ca1/kube-rbac-proxy-self/0.log" Oct 30 14:00:50.167666 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:50.167637 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-56b76dcb6d-fm86z_5b885e3c-593b-45d1-b4d4-4604ca511662/metrics-server/0.log" Oct 30 14:00:50.302644 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:50.302615 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-g2zfl_bb39553b-ccf2-4cb0-af1c-339373ec1b13/monitoring-plugin/0.log" Oct 30 14:00:51.212611 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:51.212575 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vdft6_60a88a61-3c25-48c8-84c5-b35bab4a058a/node-exporter/0.log" Oct 30 14:00:51.340071 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:51.340041 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vdft6_60a88a61-3c25-48c8-84c5-b35bab4a058a/kube-rbac-proxy/0.log" Oct 30 14:00:51.469696 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:51.469620 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vdft6_60a88a61-3c25-48c8-84c5-b35bab4a058a/init-textfile/0.log" Oct 30 14:00:51.600937 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:51.600909 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pxkqg_31e0a09f-ff45-482d-afec-c57bd2d41cb4/kube-rbac-proxy-main/0.log" Oct 30 14:00:51.727203 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:51.727115 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pxkqg_31e0a09f-ff45-482d-afec-c57bd2d41cb4/kube-rbac-proxy-self/0.log" Oct 30 14:00:51.854787 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:51.854758 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pxkqg_31e0a09f-ff45-482d-afec-c57bd2d41cb4/openshift-state-metrics/0.log" Oct 30 14:00:51.998002 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:51.997922 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f97cce5e-e85c-470a-9cd1-4b3774e90600/prometheus/0.log" Oct 30 14:00:52.128395 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:52.128368 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f97cce5e-e85c-470a-9cd1-4b3774e90600/config-reloader/0.log" Oct 30 14:00:52.257165 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:52.257080 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f97cce5e-e85c-470a-9cd1-4b3774e90600/thanos-sidecar/0.log" Oct 30 14:00:52.394174 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:52.394141 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f97cce5e-e85c-470a-9cd1-4b3774e90600/kube-rbac-proxy-web/0.log" Oct 30 14:00:52.539329 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:52.539247 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f97cce5e-e85c-470a-9cd1-4b3774e90600/kube-rbac-proxy/0.log" Oct 30 14:00:52.672690 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:52.672652 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f97cce5e-e85c-470a-9cd1-4b3774e90600/kube-rbac-proxy-thanos/0.log" Oct 30 14:00:52.801214 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:52.801138 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f97cce5e-e85c-470a-9cd1-4b3774e90600/init-config-reloader/0.log" Oct 30 14:00:53.724549 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:53.724515 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79ff6956fb-ljgd2_8915de45-eb8f-451b-8011-2cbf588de9b5/thanos-query/0.log" Oct 30 14:00:53.851254 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:53.851219 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79ff6956fb-ljgd2_8915de45-eb8f-451b-8011-2cbf588de9b5/kube-rbac-proxy-web/0.log" Oct 30 14:00:53.990236 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:53.990150 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79ff6956fb-ljgd2_8915de45-eb8f-451b-8011-2cbf588de9b5/kube-rbac-proxy/0.log" Oct 30 14:00:54.117273 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:54.117241 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79ff6956fb-ljgd2_8915de45-eb8f-451b-8011-2cbf588de9b5/prom-label-proxy/0.log" Oct 30 14:00:54.469124 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:54.469094 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79ff6956fb-ljgd2_8915de45-eb8f-451b-8011-2cbf588de9b5/kube-rbac-proxy-rules/0.log" Oct 30 14:00:54.605869 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:54.605842 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79ff6956fb-ljgd2_8915de45-eb8f-451b-8011-2cbf588de9b5/kube-rbac-proxy-metrics/0.log" Oct 30 14:00:55.161247 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:00:55.161218 7010 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rkb8w/perf-node-gather-daemonset-khfdr" Oct 30 14:00:59.400911 ip-10-0-138-103 kubenswrapper[7010]: E1030 14:00:59.400868 7010 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4hts2" containerName="extractor" Oct 30 14:01:02.997224 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:01:02.997185 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mtsz8_1069a5ce-5720-4e0d-8bb8-cc874020e3dd/dns/0.log" Oct 30 14:01:03.123514 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:01:03.123465 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mtsz8_1069a5ce-5720-4e0d-8bb8-cc874020e3dd/kube-rbac-proxy/0.log" Oct 30 14:01:03.394336 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:01:03.394285 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qldmf_109981ff-df1e-4279-bc7a-1e2b09f30ad1/dns-node-resolver/0.log" Oct 30 14:01:04.260722 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:01:04.260687 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8nb8t_e3b65b07-570d-49f3-8b74-c613b6e8c669/discovery/0.log" Oct 30 14:01:04.354342 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:01:04.354304 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65dc4d65b9-cvtrj_ea775e41-0676-42f8-937c-5b771342e724/kube-auth-proxy/0.log" Oct 30 14:01:04.677770 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:01:04.677735 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-k68w2_fd79a34d-b24a-41f2-b01a-55b70b9972f2/istio-proxy/0.log" Oct 30 14:01:04.741320 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:01:04.741293 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-sm24s_712fe9d3-fad8-4615-b9ba-18eeee7ec299/istio-proxy/0.log" Oct 30 14:01:04.871133 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:01:04.871105 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-64d4cc998c-tlqh6_3c7855fd-e6a6-44ad-870e-ed9f483528f2/istio-proxy/0.log" Oct 30 14:01:05.082085 ip-10-0-138-103 kubenswrapper[7010]: I1030 14:01:05.081989 7010 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bm8bc_8bc2e7c1-04ef-439a-90e5-57c4c200c62f/node-ca/0.log"